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1.  Introduction 


1.1  Purpose 

Small  ground  robots,  such  as  the  PackBot  and  TALON,  have  been  widely  used  by  warfighters  to 
hunt  for  terrorists  and  perform  all  types  of  reconnaissance  duties  (Axe,  2008).  Their  rugged 
small  size  and  video  capabilities  make  them  very  effective  for  non-line-of-sight  reconnaissance 
(e.g.,  search  and  assessment)  tasks  in  darkness  or  dangerous  context,  and  their  widespread  use 
reduces  the  human  risk  in  combat  reconnaissance  missions.  While  their  contributions  to  Soldier 
performance  and  well-being  have  been  established,  many  vital  robot  tasks  have  been  identified 
as  high  workload  tasks  given  Army  operational  context  (Mitchell,  2005;  Mitchell  and  Brennan, 
2009).  In  addition,  many  tasks  have  been  identified  that  can  lead  to  failure  (e.g.,  damage  to 
operator,  robot,  and/or  rescue  victim)  (Scholtz  et  al.,  2004).  Exploratory  missions  in  which  the 
robot  operator  must  teleoperate  the  robot  while  also  attending  to  the  environment  in  order  to  gain 
intelligence  result  in  especially  high  operator  workload  (Chen  et  al.,  2008).  Such  missions 
require  many  abilities,  such  as  driving,  sensing,  and  information  evaluation,  in  order  to  perform 
successfully  while  maintaining  situation  awareness  (SA). 

Improved  robot  display  and  control  capabilities  as  associated  with  telepresence  technology  have 
been  shown  to  support  performance  and  reduce  operator  workload.  Telepresence  has  been 
defined  as  “a  human-computer-machine  condition  in  which  a  user  receives  sufficient  information 
about  a  remote,  real-world  site  through  a  machine  so  that  the  user  feels  physically  present  at  the 
remote,  real-world  site”  (Aliberti  and  Bruen,  2006).  Telepresence  often  includes  capabilities  for 
a  more  naturalistic  perception  of  information,  such  as  stereo  vision,  panoramic  view,  and  stereo 
audio  (Den  Breejen  and  Jansen,  2008;  Jansen  et  al.,  2012). 

In  addition  to  a  more  naturalistic  display  of  perception  information,  telepresence  also  often 
includes  more  intuitive  control  capabilities  such  that  operator  movements  or  gestures  are 
translated  in  a  naturalistic  fashion.  The  intent  is  to  increase  ease  of  perception  and  control  and 
thus  provide  the  robot  controller  with  a  greater  ability  to  attend  to  aspects  of  SA  (Van  Erp  et  al., 
2006;  Van  Breda  and  Van  Erp,  2006).  In  this  report  we  describe  two  experiments  using 
telepresence  capabilities  for  beyond-line-of-sight  control  of  a  ground  robot  for  military 
reconnaissance  missions. 

1.2  Telepresence 

Telepresence  technology  can  enable  operator  performance  through  sensory  and  performance 
capabilities  that  allow  the  operator  to  feel  “present”  in  robot  teleoperation  tasks.  In  regular 
ground  teleoperations,  an  unmanned  ground  vehicle  (UGV)  typically  transmits  video  to  an 
operator  control  station  that  displays  the  video  on  a  computer  screen.  The  operator  then 


1 


teleoperates  the  UGV,  usually  by  using  a  joystick  or  gaming  controller  (e.g.,  handheld  button 
controls).  In  contrast,  telepresence  capabilities  incorporate  features  such  as  immersive  three- 
dimension  (3-D)  vision,  3-D  audio,  and  head-driven  camera  movement  controls.  These  features 
are  expected  to  allow  the  operator  more  seamless  perception  and  manipulation  within  the 
environment  and  higher  SA  (Van  Erp  et  al.,  2006;  Jansen  et  al.,  in  review).  Visual  search,  object 
identification,  and  object  manipulation  are  expected  to  be  improved  through  enhanced  camera 
capabilities  (e.g.,  resolution,  video  streaming,  camera  scope,  movement,  and  focus).  Normally  in 
telepresence,  the  operators  wear  a  head-mounted  display  (HMD)  and  head  tracker  to  allow  them 
to  move  the  camera  through  head  movements  as  if  they  were  seeing  through  the  robot’s  “eyes.” 
This  capability  has  also  been  demonstrated  in  a  portable  binocular  format  (Jansen,  2006).  Three- 
dimensional  audio  capability  has  been  developed  and  demonstrated  for  robot  control  tasks 
(Keyrouz  and  Diepold,  2007)  and  is  expected  to  contribute  to  performance  across  a  variety  of 
military  missions. 

Many  applications  for  telepresence  capabilities  have  been  developed  and  demonstrated. 
Telepresence  technologies  have  been  instantiated  in  medical  surgical  operations  (Alexander  and 
Maciunas,  1999;  Faust,  2007;  Haidegger  and  Benyo,  2008;  Sankaranarayanan  et  al.,  2007; 
Schostek  et  al.,  2009)  and  for  remote  patient  care  (Hu,  2008;  Schmidt  and  Holmes,  2007).  They 
have  also  been  demonstrated  in  a  broad  variety  of  settings  for  purposes  as  diverse  as  service  to 
the  public  (Gong,  2008),  control  of  humanoid  robots  (Seo  et  al.,  2006),  construction  and  vehicle 
teleoperation  (Fong  et  al.,  2001;  Sasaki  and  Kawashima,  2008),  ocean  exploration  (Manley, 

2008;  Martinez  and  Keener-Chavis,  2006),  National  Aeronautics  and  Space  Administration 
(NASA)  team  member  support  (Goza  et  al.,  2004),  museum  learning  tools  (All  and  Nourbakhsh, 
2001),  reconnaissance  and  rescue  (Zalud  et  al.,  2006),  insect  telepresence  (All  and  Nourbaskhsh, 
2001),  and  space  operations  (Foing  and  Ehrenfreund,  2008;  Fandis,  2008). 

Immersive  telepresence  has  also  been  demonstrated  for  field  operations  that  are  more  directly 
relevant  to  military  operations,  including  situations  such  as  search  and  rescue,  forestry,  mine 
operations,  remote  security  guard,  and  reconnaissance.  To  date,  results  are  mixed  but  show 
promise.  Ryu  and  colleagues  (Ryu  et  al.,  2006)  report  advancements  in  networking  that  allow 
greater  distances  in  remote  teleoperation  of  robot  security  guards.  Hai-zhou  and  his  colleagues 
(2000)  describe  the  development  of  a  telepresence  system  for  outdoor  mobile  robots,  consisting 
of  a  wide-screen  display,  remote  driving  steering  wheel,  camera  pan  and  tilt,  and  dual  3-D 
cameras. 

While  the  system  was  demonstrated  to  be  functional,  there  were  no  empirical  comparisons  of  the 
telepresence  to  a  non-telepresence  controller.  Halme  and  his  colleagues  (Halme  et  al.,  1999; 
Halme,  2010)  investigated  several  levels  of  telepresence  for  field  tasks,  ranging  from  “full” 
telepresence  (i.e.,  stereovision,  sound,  2-degree-of-freedom  [2-DOF]  head  tracking)  to  partial 
combinations:  monovision  (i.e.,  monovision,  sound,  2-DOF  head  tracking),  monitor-based  (i.e., 
image  on  screen,  sound,  2-DOF  head  tracking),  manual  (i.e.,  image  on  screen,  sound,  2-DOF 
manual  camera  control),  and  baseline  (image  on  screen,  sound,  fixed  camera).  Sound  was 
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previously  determined  to  significantly  enhance  performance.  Halme  et  al.  applied  these 
controllers  to  several  different  mission-relevant  tasks,  such  as  corridor  driving,  driving  in 
unknown  territory,  loading  tasks,  maneuvering  tasks,  off-road  driving,  and  fast  driving.  All 
conditions  were  associated  with  better  performance  than  the  condition  with  the  manual  camera 
control,  which  was  eliminated  from  further  comparisons.  The  authors  report  driving  times  were 
somewhat  shorter,  and  there  were  fewer  errors  when  using  the  head  tracker;  however,  tests  of 
significance  or  effect  sizes  were  not  reported.  There  was  some  evidence  that  the  stereo  vision 
also  contributed  to  better  performance.  Operators  could  only  accomplish  driving  in  unknown 
terrain  in  the  conditions  with  head  tracking,  as  they  could  not  recover  from  being  lost  with  only  a 
fixed  camera. 

In  general,  the  monocular  vision  performed  as  well  as  the  stereo;  however,  this  may  have  been 
because  of  the  higher  resolution  of  the  monocular  display.  Overall,  results  showed  that 
effectiveness  of  capabilities  depended  on  the  task  demands,  the  opportunities  for  training  and 
practice  (operators  improved  considerably  with  all  conditions  after  practice),  and  the  degree  of 
novelty  of  the  task  (e.g.,  operators  facing  unknown  territory  or  performing  a  task  as  a  novice). 
Generally,  the  head  tracking  was  found  to  assist  in  various  tasks,  while  stereo  vision  may  need 
further  improvements  in  resolution.  Further  research  is  ongoing  to  integrate  telepresence 
features  with  semi-autonomous  capabilities  in  various  work  operations  (Suomela  and  Halme, 
2001;  Suomela,  2004). 

Telepresence  prototypes  for  military  applications  are  being  developed  for  a  number  of  military 
purposes,  such  as  search  and  disposal  of  improvised  explosive  devices  (IEDs)  (Getlin,  2009; 
Greenemeier,  2010;  Neerincx,  in  review).  Prototypes  under  development  also  include  a  remote 
sentry  with  directional  3-D  spatial  audio  to  detect,  localize,  and  track  vehicle  targets  (Overland, 
2005;  Vaudrey  and  Sachindar,  2003).  While  these  capabilities  were  successfully  demonstrated, 
they  were  not  compared  with  any  existing  system.  In  another  effort,  immersive  telepresence  was 
expected  to  enable  higher  driving  speeds  when  compared  to  current  small  Army  UGVs 
(Yamauchi  and  Massey,  2008).  In  that  study,  the  immersive  teleoperation  was  based  on  an 
HMD  and  head-aimed  cameras.  The  combination  of  telepresence  and  semi-autonomous  obstacle 
avoidance  was  mounted  on  an  iRobot  Warrior  UGV  prototype  and  also  on  a  ruggedized  radio- 
controlled  gas-powered  car.  Researchers  reported  that  this  system  enabled  operators  to  drive  at 
full  speed  (estimated  top  speed  of  30  mph)  while  making  turns,  and  that  Soldiers  reported 
increased  SA.  However,  the  head-controlled  camera  was  not  associated  with  faster  times  than  a 
fixed-camera  view;  both  performed  very  well.  Researchers  speculated  that  the  telepresence, 
along  with  semi-autonomous  driver  assist  capability,  will  prove  more  effective  as  higher  speeds 
are  attained.  Again,  there  is  further  need  to  systematically  compare  telepresence  features,  and 
specific  combinations  of  features,  as  they  apply  to  specific  tasks. 

In  this  report  we  describe  two  experiments.  In  experiment  1,  we  focused  on  the  combination  of 
stereo  vision  and  head  tracking  capability  for  reconnaissance  tasks  that  did  not  require  3-D 
audio.  A  recent  study  investigated  the  effectiveness  of  3-D  visualization  on  robot  control 
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performance  (Pettijohn  et  al.,  2009).  In  their  study,  seven  Soldiers  performed  target  search  and 
identification  tasks  in  a  variety  of  settings  (finding  targets  buried  in  rubble  or  placed  in  a  dark 
cave,  route  reconnaissance,  etc.)  and  object  manipulation  (placing  a  chemical  sensor  very  near 
the  target,  removing  obstacles  surrounding  the  target,  etc.).  Stacked  LCD  displays  produced 
stereo  video  when  used  with  passive  polarized  glasses.  Researchers  found  that  Soldiers 
performed  search  tasks  much  faster  (22%-47%  time  savings)  with  the  3-D  system  and  reported 
that  the  3-D  system  greatly  enhanced  their  depth  perception  and  overall  performance.  Three- 
dimensional  visualization  was  also  associated  with  better  performance  and  Soldier  preference  in 
other  Army  robot  control  tasks  (Bodenhamer,  2007;  Pettijohn  et  al.,  2007).  However,  other 
results  were  mixed  with  regard  to  contribution  of  stereoscopy  on  performance;  these  results 
suggest  that  effectiveness  will  depend  on  factors  such  as  task  demands  and  image  resolution  (Lee 
and  Kim,  2008). 

For  our  first  experiment,  we  investigated  the  utility  of  the  stereo  vision  combined  with  head 
tracking  for  indoor  and  outdoor  ground  search.  For  this  task  scenario,  all  items  were  placed  on 
the  ground  to  enable  some  comparison  with  results  from  a  previous  study  using  a  similar 
baseline  system  (Pettitt  et  al.,  2010).  After  training  on  the  operation  of  the  robotic  system,  each 
Soldier  completed  indoor  and  outdoor  reconnaissance  tasks  using  the  telepresence  interface  and 
the  baseline  controller  interface  commonly  used  in  current  operations.  The  terrain  and  hazards 
were  counterbalanced  along  with  the  interface  conditions  to  control  for  practice  effects.  Display 
conditions  were  evaluated  based  on  objective  performance  data,  data  collector  observations,  and 
Soldier  questionnaires. 

From  experiment  1,  we  developed  a  plan  for  further  investigation  for  experiment  2.  For 
experiment  2,  we  created  a  more  challenging  search  environment  where  items  were  placed  above 
ground  as  well  as  on  the  ground  to  better  explore  the  utility  of  the  head-driven  camera.  We  also 
added  the  3-D  audio  capability  and  corresponding  audio  search  task  demands.  The  full 
telepresence  condition  consisted  of  stereo  vision,  3-D  audio,  and  head-tracked  camera  controller. 
The  baseline  telepresence  used  monovision,  monoaudio,  and  a  joystick  camera  controller.  A 
third  condition  used  monovision,  monoaudio,  and  the  head-tracked  camera  controller.  Soldiers 
performed  an  audio  search  task,  an  SA  search  task,  and  a  movement  control  task  where  tasks  and 
conditions  were  counterbalanced.  Measures  included  objective  performance  data,  data  collector 
observations,  and  Soldier  questionnaires. 


2.  Experiment  1 


In  this  section,  we  describe  experiment  1,  where  we  examined  operators’  experience  and 
performance  while  using  robot  telepresence  capability  during  indoor  urban  and  outdoor 
cross-country  reconnaissance  missions.  Soldier  participants  controlled  a  reconnaissance  robot, 
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developed  by  TNO  Defence,  Security  and  Safety  -  Human  Factors,  to  search  and  identify 
various  items  of  interest.  Each  Soldier  performed  equivalent  search  tasks,  once  using  a  robot 
controller  interface  with  telepresence  capability  and  once  without  the  telepresence.  This  allowed 
us  to  assess  the  relative  contribution  of  telepresence  to  robot  performance  with  regard  to 
navigation,  search  and  identification  of  items,  and  situation  assessment.  After  task  performance, 
Soldiers  provided  structured  descriptions  of  user  experiences  and  systematic  feedback  regarding 
operational  issues. 

2.1  Method 

2.1.1  Scenario 

For  this  experiment,  there  were  two  visual  search  tasks.  For  the  indoor  reconnaissance,  operators 
were  requested  to  search  each  room  in  a  building  and  locate  and  identify  each  item  of  interest. 
They  were  then  requested  to  describe  what  activity  was  most  likely  happening  and  would  likely 
occur  next,  through  inference  given  the  particular  items  found.  For  the  outdoor  reconnaissance, 
operators  were  requested  to  search  for  various  items  of  military  interest  while  navigating  a  route 
outdoors  in  relatively  rough  terrain. 

2.1.2  Participants 

Twenty  Soldiers  were  recruited  from  the  Officer  Candidate  School  (OCS)  to  participate  in  the 
study.  Solders  included  those  with  prior  service  as  well  as  those  who  entered  OCS  directly  from 
college.  Sixteen  Soldiers  were  able  to  participate  fully;  other  sessions  had  to  be  cancelled, 
primarily  because  of  weather. 

2.1.3  Equipment  and  Instruments 

The  robot  (Generaal)  used  during  the  experiment  was  developed  by  TNO  to  demonstrate 
telepresence  capabilities.  Two  interface  designs  for  robot  control  were  compared  for  navigation 
and  search  and  identification  tasks  (i.e.,  TNO  interface  design  with  and  without  the  telepresence 
features).  Routes  for  indoor  and  outdoor  reconnaissance  were  developed  to  systematically 
evaluate  robot  control  performance. 

2. 1.3.1  The  Generaal.  The  Generaal  UGV  is  a  six-wheeled  vehicle  developed  by  TNO  for 
research  and  development  purposes  (figure  1).  It  has  a  pan-tilt-roll  camera  sensor  system, 
consisting  of  stereo  vision  and  3-D  audio.  It  typically  runs  for  about  2  h,  depending  on  situation 
and  task  demands.  Sensor  and  control  signal  communication  is  IP  based  (5.8-GHz  wireless  local 
area  network),  with  an  open  development  architecture.  The  Generaal  UGV  has  ultrasonic 
collision  sensors,  with  telepresence  experience  provided  by  the  stereo  vision  and  3-D  audio.  It 
has  performed  well  in  indoor  obstacle-rich  environments. 
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Figure  1 .  The  Generaal  robot  and  a  binocular 
controller. 


2. 1.3. 2  The  Generaal  Telepresence  Controller.  The  Generaal  control  unit  (figure  2)  included 
head  tracking  equipment  that  enabled  stereo  vision  and  3-D  audio.  The  stationary  control  unit 
usually  includes  a  steering  wheel  and  foot  pedals  for  vehicle  control,  and  vehicle  rotation  was 
controlled  by  rotating  the  control  unit’s  chair.  Optional  force  feedback  on  the  operator-induced 
rotation  of  the  vehicle  can  be  provided  to  the  operator  by  corresponding  rotation  of  the  control 
unit’s  chair.  However,  in  this  experiment,  we  used  a  more  portable  laptop  system  with  a  gaming 
controller  device;  therefore,  the  telepresence  capability  was  limited  to  the  head-mounted  controls 
and  stereo  vision.  The  3-D  audio  capability  was  not  assessed  in  this  initial  experiment.  Image 
resolution  was  320  x  240  for  the  left  as  well  as  the  right  eye.  Weight  of  the  unit  was  950  g. 
Important  for  performance  is  the  cumulative  delay  in  the  whole  system.  For  this  system,  the  lag 
of  the  video  image  is  estimated  at  100-150  ms.  The  total  delay  consists  of  the  lags  in  head 
tracking  measurements,  transmission  of  those  values  to  the  robot,  turning  the  pan-tilt-roll, 
encoding  the  video  image,  sending  it  via  the  wireless  comm,  decoding  the  image  again,  and 
presenting  it  on  the  HMD.  Figure  2  shows  on  the  laptop  screen  the  same  image  as  the  Soldier 
sees  on  the  HMD. 
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Figure  2.  The  Generaal  controller  with  head-mounted 
telepresence. 

2. 1.3. 3  TNO  Baseline  Interface.  The  operator  interface  used  to  control  the  TNO  systems  was 
instantiated  on  a  laptop  and  based  on  software  that  emulates  SSC  Pacific’s  Multi-Robot  Operator 
Control  Unit  (Powell  et  al.,  2008).  An  example  screenshot  of  the  interface  is  found  in  figure  3. 
The  robot’s  location,  driven  path,  goal  points,  and  sensor  data  (i.e.,  map  data)  are  overlaid  on  an 
aerial  image.  Real-time  video  from  the  robot  is  also  displayed.  Function-button  mapping  was 
rather  simple:  one  stick  controller  for  left-right-forward -backward  control  of  the  robot,  another 
stick  controller  for  up-down-left-right  control  of  the  sensor  unit. 
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Figure  3.  Snapshot  of  laptop  interface. 

2. 1.3.4  Outdoor  Route  Reconnaissance  Course.  The  outdoor  route  reconnaissance  course 
(figures  4  and  5)  was  located  at  the  Molnar  MOUT  (Military  Operations  in  Urban  Terrain)  site  in 
Fort  Benning,  GA.  It  consisted  of  two  separate  but  equivalent  routes.  Each  route  included  three 
waypoints  and  totaled  a  length  of  180  m.  Various  types  of  IEDs,  mortar  rounds,  and  mines  that 
could  be  seen  by  the  driving  camera  were  placed  along  each  route  (figure  6).  An  equal  number 
of  small  and  large  items  were  placed  along  each  route,  with  equivalent  locations  on  the  route, 
resulting  in  items  that  were  small  and  near,  small  and  farther  away,  large  and  near,  and  large  and 
farther  away.  Operators  were  located  out  of  line  of  sight  of  the  robot  in  a  stationary  position 
inside  a  tent  located  outside  the  building.  Data  collectors  followed  behind  the  robot  as  it 
traversed  the  course.  Soldiers  were  assigned  to  devices  and  routes  in  a  counterbalanced  manner 
to  obviate  any  effects  due  to  a  particular  route. 
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Figure  4.  Outdoor  route  reconnaissance  course  diagram. 


Figure  5.  Outdoor  route. 
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Figure  6.  Examples  of  outdoor  items  of  interest. 

2. 1.3. 5  Building  Reconnaissance  Course.  The  building  reconnaissance  course  (figure  7)  was 
also  located  at  the  Molnar  MOUT  site.  For  this  experiment,  we  used  two  one-story  buildings, 
which  were  similar  in  size,  with  the  same  number  of  rooms  and  similar  floor  plans.  Rooms  in 
the  buildings  had  tables,  chairs,  and  other  furnishings.  Over  20  mock-up  IEDs,  weapons,  and 
other  items  of  interest  were  placed  along  each  building  route.  The  building  routes  differed  in 
theme  in  that  items  of  interest  indicated  a  different  purpose.  In  one  building,  the  setting 
portrayed  enemy  interrogation  of  prisoners  (i.e.,  a  prisoner-of-war  [POW]  theme);  the  other 
portrayed  enemy  plans  and  materials  related  to  a  suicide  terror  attack  on  New  York  City  (see 
items  listed  in  figure  7  and  examples  of  items  in  figure  8).  Soldiers  were  assigned  to  devices  and 
indoor  routes  to  counterbalance  the  effects  of  route  differences. 
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Sandbag  over  head 

22 
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Figure  7.  Building  reconnaissance  courses  and  items  of  interest  within  each  building. 
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Figure  8.  Examples  of  building  items  of  interest. 

2.1.4  Procedures 

2. 1.4.1  Soldier  Orientation.  The  Soldier  participants  reported  in  groups  of  four  for  1  day  each, 
from  0800  to  1800  daily.  After  arrival,  they  received  a  roster  number  for  identification 
throughout  the  evaluation.  The  Soldiers  were  oriented  on  the  purpose  of  the  study  and  their 
participation.  They  were  briefed  on  the  objectives,  procedures,  equipment,  reconnaissance 
course,  and  reconnaissance  tasks.  They  were  asked  to  review  an  operations  order  that  provided  a 
context  for  the  reconnaissance  mission.  Experimenters  answered  any  questions  the  Soldiers  had 
concerning  the  experiment.  The  Soldiers  were  also  told  how  the  results  would  be  used  and  the 
benefits  the  military  could  expect  from  the  investigation. 

2. 1.4. 2  Demographics.  Soldiers  were  requested  to  provide  demographic  data  about  their 
physical  characteristics  and  military  experience,  including  their  knowledge  of  operating 
remote-controlled  vehicles.  The  demographic  sheet  and  data  summary  are  provided  in 
appendix  A. 
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2. 1.4. 3  Training.  No  specialized  experience  was  required  from  the  requested  Soldiers.  A 
representative  from  the  U.S.  Army  Research  Laboratory  (ARL)  presented  an  overview  of  the 
experiment,  reconnaissance  courses,  and  task  demands.  Representatives  from  TNO  introduced 
the  Soldiers  to  the  use  of  the  robot  control  systems.  Soldiers  were  given  hands-on  training  on  a 
particular  interface  immediately  before  using  the  interface.  During  the  hands-on  training, 
Soldiers  performed  the  reconnaissance  tasks  on  indoor  and  outdoor  training  courses  that 
provided  practice  on  all  task  demands.  Training  evaluation  questions  were  included  in  the 
postiteration  questionnaire  regarding  adequacy  of  training  and  requests  for  comments  and 
suggestions. 

2. 1.4.4  Task  Execution.  After  Soldiers  completed  training  on  a  particular  interface,  they 
completed  the  outdoor  and  then  the  indoor  route  with  that  interface.  Soldiers  were  given  20  min 
to  complete  each  route,  and  all  performed  within  the  time  limits.  During  task  execution,  a  TNO 
experimenter  and  an  ARL  experimenter  observed  Soldier  performance,  recorded  performance 
measures,  and  noted  any  issues. 

The  entire  route  reconnaissance  course  consisted  of  the  outdoor  route  followed  by  the  indoor 
route.  There  were  two  equivalent  entire  routes;  the  Soldiers  navigated  a  different  route  with  each 
robot  interface.  They  were  instructed  to  search  for  items  of  interest  and  locate  waypoints  for  the 
outdoor  route.  The  Soldiers  were  provided  a  map  of  the  route  on  their  operator  interface  with  the 
waypoints  of  the  route  marked.  The  map  display  included  a  list  of  potential  items  that  might  be 
present  on  the  course,  but  there  was  no  indication  of  the  specific  items  that  were  placed  along  the 
route.  When  the  Soldier  found  an  item  of  interest,  he  or  she  would  tell  the  data  collector,  select 
the  item  from  the  list,  and  use  the  controller  to  take  a  picture  of  the  item.  Soldiers  drove  to  each 
waypoint  while  performing  the  search,  and  when  a  waypoint  was  reached,  they  notified  the  data 
collector.  Upon  reaching  the  last  waypoint,  the  robot  was  returned  to  the  starting  point  by  a 
TNO  experimenter. 

A  data  collector  accompanied  the  robot  to  report  start  and  stop  times,  times  off  course,  and 
driving  errors.  Another  data  collector  was  present  at  the  operator  station  to  record  the  number  of 
objects  found  and  reported  performance  measures.  Upon  completing  the  iteration  (i.e.,  an  entire 
route  reconnaissance),  the  Soldiers  moved  to  a  questionnaire  area  to  fill  out  questionnaires 
concerning  the  exercise  just  completed  (see  appendix  A)  and  to  fill  out  a  NASA  task  load  index 
(TLX)  (see  appendix  B)  concerning  the  level  of  workload  experienced. 

Once  Soldiers  completed  the  first  iteration,  including  the  questionnaires,  they  joined  the  other 
Soldiers  in  a  waiting  area.  Soldiers  were  requested,  emphatically,  to  refrain  from  discussing 
their  experience  among  themselves.  Experimenters  were  nearby  to  monitor  and  enforce  this 
request.  Soldiers  remained  in  the  waiting  area  until  they  were  called  for  their  second  iteration, 
which  proceeded  in  the  same  way  as  the  first  iteration.  After  both  iterations  were  completed, 
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Soldiers  filled  out  an  end-of-experiment  questionnaire  (see  appendix  A).  The  Soldiers  took  ~2  h 
to  complete  both  iterations,  with  all  data  collection  accomplished  in  ~8  h  in  a  single  day.  All 
Soldiers  were  available  throughout  the  8  h. 

2.1.5  Experiment  Design 

This  experiment  was  a  within-subjects  design  with  two  levels  of  robot  control  interface  (with  and 
without  telepresence).  The  order  of  treatments  and  assignment  of  Soldiers  to  routes  were 
counterbalanced  according  to  table  1.  As  the  table  shows,  Soldiers  were  processed  in  pairs.  First, 
Soldier  1  performed  iteration  1.  After  Soldier  1  completed  his  or  her  first  iteration  and  while  he 
or  she  was  providing  feedback,  Soldier  2  performed  iteration  1.  After  Soldier  2  completed  a  first 
iteration  and  while  he  or  she  was  providing  feedback,  Soldier  1  performed  a  second  iteration. 
After  Soldier  1  completed  a  second  iteration  and  while  he  or  she  was  providing  feedback,  Soldier 
2  completed  a  second  iteration  (and  so  on). 


Table  1.  Order  of  treatments  and  lanes. 


Weekday 

Roster 

(Soldier) 

First  Iteration 

Second  Iteration 

Order 

Interface 

Route 

Order 

Interface 

Route 

Monday 

1 

1 

L 

1 

3 

T 

2 

2 

2 

T 

1 

4 

L 

2 

3 

5 

L 

2 

7 

T 

1 

4 

6 

T 

2 

8 

L 

1 

Tuesday 

5 

9 

T 

1 

11 

L 

2 

6 

10 

L 

2 

12 

T 

1 

7 

13 

T 

2 

15 

L 

1 

8 

14 

L 

1 

16 

T 

2 

Wednesday 

9 

17 

L 

2 

19 

T 

1 

10 

18 

T 

2 

20 

L 

1 

11 

21 

L 

1 

23 

T 

2 

12 

22 

T 

1 

24 

L 

2 

Thursday 

13 

25 

T 

2 

27 

L 

1 

14 

26 

L 

1 

28 

T 

2 

15 

29 

T 

1 

31 

L 

2 

16 

30 

L 

2 

32 

T 

1 

Friday 

17 

33 

L 

2 

35 

T 

1 

18 

34 

T 

2 

36 

L 

1 

19 

37 

L 

1 

39 

T 

2 

20 

38 

T 

1 

40 

L 

2 

Note:  T  =  with  telepresence;  L  =  laptop  without  telepresence. 


2.1.6  Measures 

Measures  included  performance,  workload,  and  Soldier  evaluations. 

2. 1.6.1  Performance  Measures.  Performance  measures  included  the  following: 
•  Route  reconnaissance  course  completion  time 
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•  Building  reconnaissance  course  completion  time 

•  The  number  of  driving  errors  on  each  course 

•  The  type  and  number  of  correct  objects  found  on  each  course 

•  Accuracy  of  movement  to  designated  waypoints 

2. 1.6. 2  Workload.  The  NASA-TLX  requires  the  rater  to  examine  multiple  combinations  of 
dimensional  levels  (measured  on  six  subscales:  mental  demands,  physical  demands,  temporal 
demands,  own  performance,  effort,  and  frustration)  and  derive  an  overall  workload  score  (Hart 
and  Staveland,  1988).  In  addition,  the  Soldier  evaluations  included  ratings  of  difficulty  for 
specific  tasks.  Further  description  of  the  TLX  is  included  in  appendix  B.  and  the  Soldier 
evaluations  are  included  in  appendix  A. 

2. 1.6. 3  Situation  Awareness.  After  each  iteration,  Soldiers  completed  a  questionnaire  that  listed 
all  objects  in  the  building  that  were  identified  by  the  Soldier  as  recorded  by  the  data  collector. 
The  Soldiers  were  requested  to  consider  the  purpose  of  the  building,  select  the  answer  that  best 
described  that  purpose,  and  provide  a  rationale  for  their  answer,  including  a  description  of  what 
the  occupants  were  planning  to  do. 

2. 1.6.4  Soldier-Based  Evaluations.  Soldiers  completed  two  postiteration  questionnaires  and  one 
final  questionnaire.  Questionnaires  inquired  about  the  operator’s  experience  with  the  controller 
units,  overall  assessments,  and  comments.  The  questionnaires  were  designed  to  elicit  Soldiers’ 
opinions  about  their  performance  and  experiences  with  each  control  system.  The  questionnaires 
asked  the  Soldiers  to  rate  the  devices  on  a  seven-point  semantic  differential  scale  ranging  from 
“extremely  good/easy”  to  “extremely  bad/difficult.”  Copies  of  each  questionnaire  are  included 
in  appendix  A. 

2.1.7  Data  Analysis 

Descriptive  statistics  were  calculated  for  each  measure.  Performance  data  was  analyzed  using 
repeated  measures  analysis  of  variance  (ANOVA).  Follow-on  pairwise  comparisons  were 
calculated  using  Holm’s  Bonferroni  procedure  (Holm,  1979)  to  control  for  family-wise  error 
rates.  Partial  eta  squared  (rfp),  an  index  of  effect  size,  was  computed  for  each  ANOVA. 


* 

Holm’s  Bonferroni  correction:  The  p  value  required  for  significance  is  based  on  the  number  of  t-tests  calculated.  Take  the 
r-test  with  the  smallest  p  value  and  divide  your  alpha  (typically  .05)  by  k,  the  number  of  paired  comparisons.  The  partial  eta 
squared  is  the  proportion  of  the  the  effect  +  error  variance  that  is  attributable  to  the  effect. 
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2.2  Results 


2.2.1  Participant  Demographics 

Twenty  Soldiers  were  recruited  from  the  OCS  at  Fort  Benning.  Of  these,  complete  data  were 
collected  on  16  Soldiers.  Soldiers  were  primarily  male  (two  were  female).  The  average  age  was 
25.  Nine  of  the  Soldiers  had  previous  military  experience,  averaging  8  months.  All  reported 
familiarity  and  daily  use  of  computers.  Two  reported  previous  experience  with  electronic 
military  displays,  and  three  reported  experience  with  (civilian)  robotic  systems.  Most  reported 
low  (none  or  beginner)  experience  for  operating  robot  systems.  Detailed  results  are  listed  in 
appendix  A. 

2.2.2  Training 

Soldiers  reported  high  satisfaction  with  all  aspects  of  training.  The  mean  ratings  regarding 
training  are  provided  in  table  2.  Ratings  (seven-point  scale)  ranged  from  5.81  (outdoor 
reconnaissance)  to  6.44  (overall  concept).  Detailed  results  are  provided  in  appendix  A.  Soldiers 
reported  that  the  easiest  tasks  to  learn  were  tasks  associated  with  the  camera  controls  and  driving. 
The  more  difficult  tasks  were  associated  with  coordination  of  driving,  searching,  and  taking 
pictures.  Also,  Soldiers  reported  that  using  specific  camera  movement  controls  on  both  systems 
was  more  difficult  to  learn. 

Table  2.  Mean  ratings  for  aspects  of  training. 


Training 

Mean  Response 

No.  of 
Soldiers 

Telepresence 

Laptop 

Completeness  of  introductory  training 

16 

6.31 

6.31 

Comprehension  of  overall  concept  of  the  robot 

16 

6.31 

6.44 

Outdoors:  How  to  drive 

16 

6.19 

6.25 

Outdoors:  Time  provided  to  practice 

16 

6.13 

6.06 

Outdoors:  How  to  perform  reconnaissance 

16 

5.81 

6.13 

Building:  How  to  drive 

16 

6.19 

6.19 

Building:  Time  provided  to  practice 

16 

6.06 

6.00 

Building:  How  to  perform  reconnaissance 

16 

6.13 

6.25 

Overall  training  evaluation 

16 

6.13 

6.38 

2.2.3  Performance  Outcomes 

2.2.3. 1  Performance  on  Building  Reconnaissance.  Overall,  Soldiers  performed  well  using  either 
system  when  performing  indoor  reconnaissance. 

•  Number  of  items  detected.  The  mean  number  of  items  found  for  the  baseline  laptop 
condition  was  22.19  (standard  deviation  [sd]  =  1.05)  compared  to  21.75  (sd  =  1.00)  for 
telepresence.  This  difference  was  not  statistically  significant  (F  =  1.05;  df  15,  p  =  0.32, 
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rfp  =  0.06).  Performance  also  did  not  vary  between  the  two  buildings.  With  the  laptop 
system,  the  means  were  22.22  (bldg.  1)  and  21.87  (bldg.  2);  with  the  telepresence  system, 
means  were  21.25  (bldg.  1)  and  22.25  (bldg.  2). 

•  Number  of  items  correctly  identified.  The  mean  number  of  items  correctly  identified  for 
the  laptop  was  20.62  (sd  =  1.82)  compared  to  20.12  (sd  =  2.65)  for  telepresence.  This 
difference  was  not  statistically  significant  (F  =  0.38;  df  15,  p  =  0.55,  rfp=  0.025). 

•  Time  to  complete.  Soldiers’  time  to  complete  the  building  reconnaissance  was  not 
significantly  different  between  laptop  and  telepresence  conditions  (F  =  0.34,  df  =  15, 

p  =  0.57,  rfp  =  0.22).  The  mean  time  for  the  laptop  was  9.39  min  (sd  =  2.49)  and  10.03 
min  (sd  =  1.34)  for  the  telepresence  system. 

•  Driving  errors.  The  number  of  driving  errors  within  the  building  was  not  significantly 
different  between  the  laptop  and  telepresence  conditions  ( F  =  0.01,  df  =  15,  p  =  0.91, 
rfp  =  0.001).  The  mean  number  of  errors  for  the  laptop  was  2.31  (sd  =  2.02)  and  2.38 
(sd  =  1.41)  for  the  telepresence  system. 


Figure  9  provides  summary  results  for  the  building  reconnaissance  performance  measures. 


Mean  Performance  Measures:  Building 
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Figure  9.  Mean  measures  of  building  reconnaissance  performance  for  laptop  and  telepresence  conditions. 

2. 2. 3. 2  Performance  on  Outdoor  Reconnaissance.  Overall,  Soldiers  performed  well  using  either 
system  when  performing  outdoor  reconnaissance. 
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•  Number  of  items  detected.  The  mean  number  of  items  detected  for  the  laptop  system  was 
5.37  (sd  =  2.09)  and  for  telepresence,  4.50  (sd  =  1.63).  The  difference  approached 
significance  (F  =  3.69,  df  =  15,  r|  p=0.20,  p  =  0.07).  It  should  be  noted  that  the  eta-square 
effect  size  value  is  considered  moderate  (from  0.20  to  0.40). 

•  Number  of  items  correctly  identified.  Data  collectors  recorded  the  number  of  items  that 
were  detected  but  incorrectly  identified.  Mean  values  were  very  low  and  equivalent  among 
the  two  systems;  the  laptop  was  0.50  (sd  =  0.73)  and  telepresence  was  0.50  (sd  =  1.03). 

•  Number  of  waypoints.  This  is  the  number  of  waypoints  that  the  operator  correctly 
navigated  and  identified.  For  the  laptop  system,  the  mean  was  2.68  (sd  =  0.48);  for 
telepresence,  the  mean  was  2.62  (sd  =  0.50).  This  difference  was  not  significant  (F  =  0.14, 
df=  15,  q2p  =  0.00;  p  =  0.72). 

•  Time  to  complete.  The  mean  time  to  complete  the  outdoor  course  with  the  laptop  was 
9.24  min  (sd  =  3.24);  the  mean  time  for  telepresence  was  9.45  min  (sd  =  2.57).  This 
difference  was  not  significant  (F  =  0.16,  df  =  15,  rfp  =  0.01  ,p  =  0.69). 

•  Driving  errors.  There  were  no  outdoor  driving  errors  reported  for  either  system  condition. 


Figure  10  provides  summary  results  for  the  outdoor  reconnaissance  performance. 


Mean  Performance  Measures:  Outdoors 
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Figure  10.  Mean  measures  of  outdoor  reconnaissance  performance  for  the  laptop  and 
telepresence  conditions 


2.2.4  Soldier-Based  Assessments 

2.2.4. 1  Postiteration  Soldier-Based  Assessments.  Means  for  all  ratings  of  the  laptop  and 
telepresence  conditions  for  outside  and  inside  reconnaissance  conditions  are  provided  in 
appendix  A.  Ratings  were  grouped  and  averaged  for  particular  functions.  For  example,  items 
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pertaining  to  moving  or  controlling  the  camera  were  clustered  to  obtain  overall  ratings  for 
camera  control.  Table  3  provides  means  for  these  item  groupings. 


Table  3.  Mean  Soldier  ratings  for  telepresence  and  laptop  systems  by  indoor  and  outdoor  task 
context. 


Task 

No.  of 
Soldiers 

Mean  Response 

Outdoors 

Indoors 

Tele 

Laptop 

Tele 

Laptop 

Camera  control 

16 

5.22 

5.12 

5.10 

5.14 

Moving  the  robot 

16 

4.81 

5.64 

5.41 

5.59 

Visual  identification 

16 

4.64 

5.05 

5.06 

5.38 

Overall  Assessments 

Finish  the  course  quickly 

16 

4.75 

5.19 

5.44 

5.25 

Find  improvised  explosive  devices 

16 

4.25 

4.94 

5.94 

6.31 

Maintain  situation  awareness 

16 

5.25 

5.25 

5.25 

5.12 

Overall  ability  to  perform  this 
reconnaissance 

16 

4.94 

5.62 

5.56 

5.64 

Results  appear  to  indicate  a  general  trend  in  favor  of  the  laptop;  however,  the  differences  were 
generally  small.  Only  the  difference  between  telepresence  and  laptop  for  moving  the  robot 
outdoors  was  statistically  significant  (p  =  0.026).  In  addition,  all  ratings  were  averaged  to 
produce  an  overall  mean  for  each  Soldier  for  each  condition  (i.e.,  laptop-outside,  laptop-building, 
telepresence-outside,  telepresence-building).  The  repeated  measures  general  linear  model 
resulted  in  an  F  statistic  of  2.93  (df  =  3,  rfp  =  0. 16,  p  =  0.04);  the  means  are  portrayed  in 
figure  11.  Follow-on  paired  comparison  tests  showed  that  no  particular  comparison  was 
statistically  significant. 
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Figure  11.  Mean  overall  ratings  for  laptop  and  telepresence  for  indoor  and  outdoor  performance. 
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2. 2. 4. 2  Soldier  Reports  of  Discomfort.  Table  4  provides  a  summary  of  Soldier  reports  of 
various  symptoms  of  discomfort.  Telepresence  was  associated  with  a  higher  number  of  reported 
symptoms,  particularly  for  eyestrain.  Comments*  regarding  the  telepresence  display  consisted  of 
dry  eyes  (1),  poor  image  quality  (3),  tunnel  vision  (1),  camera  alignment  (1),  slight  motion 
sickness  (1),  and  “a  little  blurry  for  the  first  5  seconds”  (1). 

Table  4.  Summary  of  reports  of  discomfort. 


Symptoms 

No.  of  Responses 

Telepresence 

Laptop 

Eyestrain 

9 

3 

Tunnel  vision 

2 

0 

Headaches 

0 

0 

Motion  sickness 

3 

1 

Nausea 

1 

0 

Disorientation 

4 

2 

Dizziness 

0 

0 

Competition  between  eyes  for  vision 

4 

0 

Any  other  problems? 

2 

0 

2. 2. 4. 3  SA  Measure.  Soldiers  responded  to  questionnaire  items  that  asked  their  opinion  as  to 
what  was  going  on  in  the  buildings  and  which  items  led  them  to  their  conclusions.  These  items 
were  used  to  assess  three  factors  of  their  SA:  perception,  comprehension,  and  projection 
(Endsley,  1995).  Results  are  provided  by  building  because  each  building  had  a  different  set  of 
items  and  a  different  theme. 

•  Perception.  First,  it  should  be  noted  that  the  primary  indicator  of  SA,  attributable  to  the 
robot  and  display  capabilities,  is  the  number  of  items  that  were  detected  and  correctly 
identified.  As  summarized  previously  in  the  performance  measures  section,  all  Soldiers 
performed  well,  with  little  difference  between  laptop  and  telepresence.  The  mean  number 
of  items  found  for  the  laptop  system  was  22.19  (sd  =  1.05)  compared  to  21.75  (sd  =  1.00) 
for  telepresence.  Performance  did  not  vary  between  the  two  buildings.  Also,  the  mean 
number  of  items  correctly  identified  for  the  laptop  was  20.62  (sd  =  1.82)  compared  to 
20.12  (sd  =  2.65)  for  telepresence.  This  difference  was  not  statistically  significant  (F  = 
0.38;  df  15;  q2p  =  0.025;  p  =  0.55). 

•  Comprehension.  Soldiers  were  good  at  diagnosing  the  building  situations  and  providing  a 
plausible  rationale  (i.e.,  comprehension).  Soldiers  were  asked  what  they  thought  was 
happening  inside  the  building  before  the  reconnaissance.  For  the  building  with  the  POW 
theme,  100%  of  the  Soldiers  responded  correctly  (e.g.,  relevant  to  POWs)  regardless  of 
display  condition.  Soldiers  were  less  in  agreement  with  regard  to  the  IED  building:  only 


The  number  of  Soldier  reports  is  provided  after  each  comment. 
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62%  of  the  Soldiers  provided  the  subject-matter  expert  (SME)-chosen  response  (i.e., 
relevant  to  a  terror  attack);  however,  most  of  their  responses  were  similar  in  theme. 

•  Projection.  Soldiers  were  also  asked  what  they  thought  would  occur  in  the  building  in  the 
near  future  (i.e.,  projection)  with  regard  to  the  enemy  plans.  For  the  question  regarding 
future  intent,  Soldiers  were  asked  to  provide  two  responses.  For  the  building  with  the 
POW  theme,  86%-89%  of  the  Soldiers  provided  a  correct  response,  with  no  difference 
between  display  conditions.  For  the  building  with  the  IED  theme,  88%  of  the  Soldiers 
provided  at  least  one  correct  response. 

Soldiers  were  also  asked  which  items  in  the  building  led  them  to  their  conclusion.  These 
responses  were  compared  to  the  list  of  items  identified  by  an  SME.  There  was  more 
variance  in  these  responses;  67%  of  the  Soldiers  agreed  with  the  SME  in  the  laptop 
condition,  and  72%  agreed  with  the  SME  in  the  telepresence  condition  for  the  POW- 
themed  building.  Soldiers  agreed  on  certain  core  items  (e.g.,  tortured  dead  body,  bloody 
rags,  bedding,  battery,  jumper  cables)  but  varied  in  their  selection  and  interpretation  of 
other  items.  For  the  IED  building,  56%  of  Soldiers  agreed  with  the  SME  in  the 
telepresence  condition  compared  to  65%  in  the  laptop  condition.  Most  Soldiers  did  provide 
a  plausible  rationale  for  their  particular  selection  of  items.  For  example,  while  “cardboard 
box”  was  not  identified  as  critical  to  the  correct  situation  assessment,  those  Soldiers  who 
did  select  the  box  as  a  relevant  item  also  inferred  that  the  box  represented  an  intent  to  pack 
and  move.  Similarly,  some  Soldiers  inferred  from  the  presence  of  weapons  and  arms  that 
the  enemy  was  surprised  and  would  return  to  the  building  later. 

2. 2. 4.4  End-of-Experiment  Soldier-Based  Evaluations  of  Controller  Characteristics.  Mean 
Soldier  evaluations  of  controller  characteristics  are  provided  in  table  5.  In  general,  the  laptop 
had  slightly  higher  ratings  for  display  characteristics,  while  the  telepresence  condition  had  some 
slightly  higher  ratings  for  controller  features.  However,  these  differences  are  not  likely  to  be 
significant. 

In  order  to  perform  tests  of  significance,  ratings  pertaining  to  displays  and  to  controller  features 
were  averaged  separately  (figure  12).  Repeated  measures  ANOVA  analyses  were  not  significant 
for  the  display  ratings,  F  =  1.050  (df  =  16,  q2p  =  0.06,  p  =  0.32),  or  for  the  controller  ratings, 

F  =  0.02  (df  =  16,  q2p,  =  0.00,  p  =  0.88). 
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Table  5.  Mean  ratings  of  controller  characteristics. 


Controller  Characteristics 

Mean  Response 

No.  of 
Soldiers 

Telepresence 

Laptop 

Resolution  (clarity) 

16 

5.25 

5.81 

Size  of  objects  appearing  in  the  display 

16 

5.25 

5.38 

Ability  to  adjust  display 

15 

4.60 

4.87 

Comfort  of  viewing 

16 

4.75 

5.81 

Display  brightness 

16 

5.25 

5.81 

Display  glare 

16 

5.63 

5.13 

Contrast  between  objects  on  the  driving  display 

15 

5.53 

5.47 

Display  color 

15 

5.47 

5.67 

Comfort  of  using  display 

16 

5.31 

5.69 

Number  of  controls 

16 

5.88 

5.50 

Control  locations 

16 

6.25 

5.94 

Size  of  individual  controls 

16 

6.25 

6.13 

Complexity  of  controls 

16 

6.25 

5.88 

Ability  to  use  controls  without  activating  other  controls 

16 

5.94 

6.12 

Size  of  entire  control  unit 

16 

6.56 

6.56 

Adequacy  of  this  control  unit  for  teleoperation 

16 

6.00 

6.20 

Ease  of  controlling  camera  (viewing  direction) 

16 

5.38 

5.31 

Overall  assessment  of  this  control  unit 

16 

5.75 

5.81 

Figure  12.  Means  of  averaged  ratings  for  display  and  controller  features. 
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2. 2. 4. 5  End-of-Experiment  Soldier-Based  Evaluations  for  Task  Performance.  Mean  ratings  for 
ease  of  performing  tasks  are  provided  in  table  6.  In  general,  ratings  were  slightly  higher  for  the 
laptop,  with  the  exception  of  overall  ratings  for  building  reconnaissance. 

Table  6.  Mean  ratings  for  ease  of  performing  tasks. 


Task 

Mean  Response 

No.  of 
Soldiers 

Telepresence 

Laptop 

Avoiding  obstacles 

16 

5.31 

6.13 

Assessing  terrain  for  navigability 

16 

5.81 

6.06 

Driving  straight  route 

16 

5.25 

6.13 

Driving  multiple  waypoints 

16 

5.44 

6.06 

Looking  for  objects  of  interest 

16 

5.50 

5.69 

Identifying  objects 

16 

5.44 

5.75 

Scanning  surroundings 

16 

5.75 

5.56 

Maneuvering  corners 

16 

5.50 

5.88 

Maintaining  situation  awareness 

16 

5.31 

5.87 

Overall  building  reconnaissance 

16 

5.81 

5.63 

Overall  outdoor  reconnaissance 

16 

5.06 

6.00 

Ratings  were  averaged  separately  for  the  laptop  and  telepresence  conditions.  The  mean  rating 
for  laptop  was  5.82  (sd  =  0.61)  and  telepresence  was  5.43  (sd  =  0.99).  This  difference  was  not 
significant  (F  =  1.76,  df  =  16,  rfp=  0.10,  p  =  0.20). 

2. 2. 4. 6  Workload:  NASA-TLX.  Mean  ratings  for  the  NASA-TLX  workload  scales  are 
provided  in  figure  13.  Paired  /-test  comparisons  were  significant  for  effort  (telepresence  was 
significantly  higher  at  p  =  0.005)  and  total  workload  (telepresence  mean  of  52.0  was  significantly 
higher  than  the  laptop  mean  of  42.53,  at  p  =  0.05). 


23 


60 


Telepresence 


La  ptop 


frustration 


9.14 


8.88 


effort 


12.53 


8.1 


performance 


7.75 


7.63 


tempo 


7.84 


8.84 


physical 


3.24 


0.29 


mental 


11.51 


8.78 


■  frustration 

■  effort 

■  performance 

■  tempo 

■  physical 

■  mental 


Figure  13.  Mean  NASA-TLX  workload  ratings. 

2.2.5  Soldier  Comments  and  Suggestions 

Soldiers  provided  numerous  comments  throughout  data  collection  in  response  to  open-ended 
questions.  All  comments  are  listed  in  appendix  A,  organized  by  questionnaire.  The  following 
tables  consolidated  comments  across  all  questionnaires  in  order  to  categorize  the  comments  with 
regard  to  (a)  favorable,  (b)  concerns,  and  (c)  suggestions  for  improvement. 

2.2.5. 1  Favorable  Comments.  Table  7  provides  favorable  Soldier  comments  in  general  order  of 
the  number  of  comments. 
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Table  7.  Summary  of  favorable  comments. 


Comment  Category 

No.  of 
Comments: 
Telepresence 

No.  of 
Comments: 
Laptop 

Easy  to  use  (in  general). 

11 

16 

Overall  system  (great  system,  etc.). 

6 

2 

Easy  to  use  indoors. 

9 

2 

Easy  to  see  indoors. 

7 

3 

Easy  to  see  through  robot  eyes. 

6 

0 

Easy  to  maneuver. 

3 

2 

Easy  to  maneuver  indoors. 

2 

0 

Easy  to  use  outdoors. 

4 

3 

Easy  to  identify  objects. 

2 

0 

Easy  to  use  camera. 

2 

1 

Red  and  green  lines  make  it  easy  to  orient  camera. 

0 

1 

Re-center  button  is  very  helpful  for  quick  correction. 

0 

1 

Controls  sensitive,  responsive. 

2 

0 

Navigation  easier  after  practice. 

2 

1 

Telepresence  more  effective  indoors,  particularly  to  maneuver 
and  have  situation  awareness. 

2 

0 

Movement  speed  of  the  robot  was  excellent  (easy  to  control; 
responsive). 

1 

0 

Stereo  vision  was  helpful. 

2 

0 

Controls  easy  to  use. 

1 

0 

Map  display  very  useful. 

0 

1 

Prefer  controller  to  telepresence. 

0 

1 

The  “1”  button  to  re-center  the  eyes  to  the  body  was  very  useful. 

0 

1 

More  comfortable,  felt  less  time  pressure. 

1 

0 

Better  for  indoors. 

2 

0 

Better  for  outdoors. 

0 

1 

Better  for  observations. 

2 

0 

More  familiar. 

0 

1 

2. 2. 5. 2  Concerns.  Table  8  provides  Soldier  comments  of  concern  in  general  order  of  the  number 
of  comments. 


25 


Table  8.  Summary  of  concerns. 


Comment  Category 

No.  of 
Comments: 
Telepresence 

No.  of 
Comments: 
Laptop 

Video  resolution  (poor,  lower,  etc.). 

11 

4 

Slight  disorientation  during  quick  head  movements;  head  movements 
difficult  while  moving  quickly;  camera  jumped  around;  difficult  to 
adjust  camera;  hard  to  keep  camera  steady. 

7 

0 

Camera  moves  too  quickly,  jerky,  would  freeze. 

0 

6 

Headpiece  bulky  and  rear  cords  constrained  movement. 

4 

0 

Difficult  to  maneuver  and  search  at  same  time. 

0 

3 

Nausea. 

3 

0 

Camera  controls  too  sensitive. 

0 

2 

Hard  to  see  when  looking  down  at  my  feet. 

2 

0 

Lag  between  operator  and  camera  movements. 

0 

2 

Difficult  to  maneuver  robot  indoors  around  turns. 

4 

0 

Rough  terrain  made  movement  difficult. 

2 

0 

Not  for  urban  ops. 

2 

0 

Lag  between  operator  and  robot  motions. 

1 

0 

Camera  height  (too  high)  hard  to  know  if  on  course. 

1 

0 

Robot  seemed  to  veer  when  moving  forward  and  looking  around  at 
same  time. 

1 

0 

Navigation  with  headset  required  switching  back  to  map  to  drive  to 
waypoints,  but  then  can’t  see  IEDs. 

1 

0 

Alignment  of  head  movement  and  camera:  “The  alignment  of  me 
facing  forward  with  the  robot  eyes  facing  forward  with  the  robot  body 
was  gradually  drifting  out  of  alignment.  This  meant  that  after  some 
time,  I  would  face  front,  but  the  robot  was  not  facing  straight  ahead  but 
off  to  the  side,  and  I  would  have  to  re-calibrate  the  robot  to  face  front.” 

1 

0 

Robot  kept  spinning  in  circles  on  its  own. 

1 

0 

Camera  would  freeze  and  system  rebooted. 

1 

0 

Had  to  be  recalibrated. 

1 

0 

Commands  entered  (such  as  “view  satellite  map”  button)  wouldn’t 
work. 

1 

0 

Difficult  to  maintain  SA  outdoors  while  moving.  Controls  did  not  feel 
smooth. 

0 

1 

Controls  confusing,  hard  to  keep  track  of  each  button  function. 

0 

1 

Smaller  size  of  rooms,  thus  more  difficult  to  maneuver  indoors. 

1 

0 

Rooms  seemed  smaller  than  they  really  were. 

1 

0 

Taking  pictures  hard,  because  had  to  switch  screens  and  select  a 
category. 

1 

0 

Robot  bulky,  hard  to  maneuver. 

1 

0 

Bumpy  terrain  contributed  to  slight  motion  sickness,  resolved  by  mono. 

1 

0 

Don’t  need  stereo,  mono  is  good. 

1 

0 

Laptop  map  good  for  outdoors  recon. 

0 

1 

Controller  and  display  did  not  feel  like  one  unit. 

0 

1 

Sometimes  hard  to  see  antennas,  made  it  hard  to  navigate  turns. 

1 

0 

Too  small  (display). 

0 

1 

Waypoint  ID  difficult  because  of  two  maps  (toggling  between). 

1 

0 
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2. 2. 5. 3  Suggestions.  Table  9  provides  Soldier  comments  of  suggestions  in  general  order  of  the 
number  of  comments. 


Table  9.  Summary  of  suggestions. 


Comment  Category 

No.  of 
Comments: 
Telepresence 

No.  of 
Comments: 
Laptop 

Needs  bigger  video  and  smaller  map. 

0 

12 

Increase  size  of  camera  display. 

0 

7 

Improve  resolution. 

8 

5 

Needs  zoom  function. 

4 

6 

Put  a  small  map  display  at  the  bottom  of  the  telepresence  screen. 

5 

0 

Invert  the  Y  axis  of  the  camera  controls. 

0 

4 

Need  ability  to  customize  screen,  change  size  of  displays,  etc. 

0 

2 

Use  a  head-mounted  sensor  to  direct  the  robot’s  eyes,  but  let  the  user 
view  the  video  (mono)  on  a  laptop  screen.  Resolution  was  much  better 
on  the  screen. 

2 

0 

Would  like  more  control  of  volume  on  the  handset,  especially  when 
going  over  rocks. 

1 

0 

Prevent  video  freezing. 

1 

0 

Fix  camera  drift,  so  that  forward  is  always  forward. 

2 

0 

It  would  help  a  lot  if  I  could  make  the  robot  have  slight  turns  of  its  head 
at  first.  Then  its  head  moves  faster  as  I  push  the  controller  harder  or  hold 
it  down  longer. 

0 

1 

Adding  360°  ability  would  enable  the  robot  to  navigate  around  turns 
better.  Hard  to  judge  how  far  you  can  swing  out. 

0 

2 

Add  rear  camera. 

0 

1 

The  camera  -  there  needs  to  be  a  sound  effect  or  some  kind  of  indication 
or  flashing  light  or  something  to  confirm  I  took  a  picture.  Sometimes  a 
small  image  of  the  photo  I  took  would  appear,  and  when  there  were 
several  pictures  to  take  at  once  it  was  unclear  if  I  got  the  picture  or  if  the 
button  didn’t  push  or  what. 

0 

1 

Make  camera  control  less  sensitive. 

1 

1 

Have  camera  higher  off  the  ground. 

0 

1 

Make  camera  less  jumpy  when  head  is  moved. 

1 

0 

Add  option  like  tapping  the  “eyes  left”  stick  twice  to  make  the  robot  look 
90°  left. 

0 

1 

Make  predefined  functions,  such  as  “right  face,”  "left  face,”  etc. 

0 

1 

Mount  from  ceiling. 

1 

0 

Attach  the  cord  so  it  does  not  slip  or  get  caught. 

1 

0 

Put  list  of  objects  on  both  views. 

1 

0 

More  contrast  between  colors. 

1 

0 

Wider  peripheral  view. 

1 

0 

Lighter  headpiece. 

1 

0 

Don’t  need  stereo. 

1 

0 

Add  keyboard  for  control  (as  well  as  joystick). 

0 

1 
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2. 2. 5. 4  Overall  Preference.  Table  10  provides  the  number  of  Soldiers  preferring  the 
telepresence  or  the  laptop  system  overall.  While  Soldiers  had  some  concerns  with  the 
telepresence  and  often  rated  it  less  highly  for  current  characteristics  and  capabilities,  9  of  the  17 
Soldiers  preferred  the  telepresence  display. 


Table  10.  Number  of  Soldiers  preferring  each  system. 


No.  of  Responses  in  Favor  of: 

Telepresence 

Laptop 

9 

6 

2.3  Summary  and  Discussion 

In  this  preliminary  investigation,  we  explored  the  relative  utility  of  two  telepresence  capabilities 
(head  tracking  and  stereo  vision)  for  indoor  and  outdoor  reconnaissance  tasks.  Two  robot 
controllers,  developed  by  TNO,  were  compared:  (1)  a  baseline  laptop  controller  that  used  a 
gaming  joystick  for  robot  and  camera  control  and  (2)  a  partial  telepresence  controller  that  used  a 
head-mounted  camera  controller  and  stereo  vision. 

Overall  results  showed  little  difference  in  task  performance  measures  between  the  two 
controllers.  Soldiers  performed  well  with  both  types  of  controllers  for  detection  and 
identification  of  items,  driving  performance,  and  driving  speed  (i.e.,  course  completion  time). 
Soldier  evaluations  were  also  similar,  rating  both  systems  favorably.  There  were,  however,  more 
reports  of  Soldier  eyestrain  with  the  telepresence  system  and  higher  values  for  estimates  of 
workload.  Reports  of  eyestrain,  headache,  or  motion  sickness  have  been  associated  with 
telepresence,  similar  to  experiences  of  virtual  reality,  and  compounded  by  technical  issues,  such 
as  camera  lag  or  interocular  adjustments  regarding  the  stereo  display  (Van  Erp  et  al.,  2006). 
Soldiers  commented  that  the  visual  display  was  not  as  clear  with  the  stereo  vision  (telepresence 
system)  compared  to  the  laptop,  and  that  they  preferred  the  higher  clarity  of  the  mono  display 
(laptop  system).  They  also  commented  that  the  telepresence  system  was  heavy  and  bulky  on  the 
head. 

While  Soldiers  performed  well  with  both  controllers,  they  also  provided  many  comments  with 
regard  to  how  each  system  could  be  improved.  All  of  the  Soldiers  suggested  that  the  laptop 
controller  have  a  larger  camera  display  to  take  up  more  of  the  laptop  screen.  In  this  study,  the 
4-in  camera  screen  size  was  made  to  be  equivalent  to  a  U.S.  laptop-based  robot  controller  display 
(i.e.,  the  MOCU,  multirobot  operator  control  unit).  Other  common  suggestions  were  to  improve 
resolution  and  add  a  zoom  function  to  both  displays.  In  addition,  they  suggested  adding  a  map  to 
the  telepresence  display  and  inverting  the  Y  axis  of  camera  controls  for  the  laptop  display.  It 
should  be  noted  that,  comments  withstanding,  more  of  the  Soldiers  preferred  the  telepresence 
controller  to  that  of  the  laptop.  Comments  appeared  to  acknowledge  the  potential  of  the 
telepresence  approach  once  certain  improvements  are  made  to  it. 
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As  a  result  of  this  study,  several  improvements  have  been  made  to  the  telepresence  controller. 
The  synchronization  between  left  and  right  stereoscopic  channels  has  been  improved.  In 
addition,  the  visual  quality  for  each  channel  has  a  higher  pixel  resolution.  TNO  is  also 
improving  procedures  to  verify  depth  perception. 

In  the  next  experiment  (2),  task  demands  will  be  tailored  to  better  assess  telepresence 
capabilities.  For  example,  in  the  current  experiment,  the  visual  search  task  included  only  items 
placed  on  the  ground.  To  better  investigate  ease  of  camera  control,  we  will  place  items  in  a 
variety  of  locations  such  that  the  robot  camera  will  have  to  look  up,  around,  and  behind  objects. 
In  addition,  we  will  include  a  visual  tracking  task,  where  the  robot  controller  will  be  asked  to 
track  and  identify  the  movements  of  a  human  confederate  who  will  move  about  while  placing 
and  picking  up  items.  The  robot  controller  will  have  to  describe  these  movements  and  behaviors. 
The  stereo  vision  capability  will  be  further  evaluated  through  an  object  manipulation  task.  In 
addition,  task  demands  will  also  evaluate  3-D  audio  relative  to  mono  audio  capabilities;  the  robot 
controller  will  be  asked  to  quickly  locate  audio  signals. 


3.  Experiment  2 


Experiment  2  builds  upon  experiment  1  through  the  investigation  of  stereo  vision  and  head 
tracking  capabilities  for  a  visual  search  task.  In  this  experiment,  the  visual  search  task  included 
items  that  were  located  above  the  robot  as  well  as  items  placed  on  the  ground.  In  addition,  3-D 
audio  capability  was  also  compared  to  mono  audio  for  an  audio-based  target  localization  task. 
While  search  tasks  are  primarily  visual  in  cognitive  demand,  in  combat  situations  there  are  many 
vital  cues  that  are  auditory  in  nature  and  often  indicate  a  threat  or  need  for  attention. 

3.1  Method 

3.1.1  Scenario 

For  this  experiment,  we  developed  two  indoor  reconnaissance  tasks.  In  the  first  reconnaissance 
task,  we  placed  various  items  of  military  interest  (e.g.,  various  ordinances  as  in  experiment  1) 
within  an  indoor  environment  simulating  a  military  room-clearing  task.  Items  were  placed  in 
various  locations,  on  the  ground  and  also  above  the  robot.  For  this  task,  we  associated  particular 
items  with  an  audio  cue.  For  each  search  task,  a  single  audio  cue  was  activated.  The  operator 
was  required  to  search  for  the  particular  item  associated  with  the  audio  cue  as  quickly  as 
possible.  They  were  then  instructed  to  approach  the  item  as  close  as  possible  without  disturbing  it. 

In  a  second  task,  various  items  were  placed  in  the  room,  some  of  which  were  distinguished  with 
a  distinct  audio  cue.  For  this  task,  operators  were  requested  to  search  for  all  items  and  build  a 
mental  map  of  as  many  items  as  they  could  remember.  They  were  then  requested  to  indicate,  on 
a  paper  map,  which  items  they  remembered  in  particular  locations. 
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3.1.2  Participants 


Twenty-two  Soldiers  were  recruited  from  the  OCS  to  participate  in  the  study.  Solders  included 
those  with  prior  service  as  well  as  those  who  entered  OCS  directly  from  college.  Eighteen 
Soldiers  were  able  to  participate  fully;  other  sessions  had  to  be  cancelled,  two  due  to  time 
constraints  and  two  due  to  discomfort  (e.g.,  nausea). 

3.1.3  Equipment  and  Instruments 

3. 1 .3. 1  The  Generaal  UGV.  The  same  robot  used  during  experiment  1  (Generaal)  was  used  in 
this  experiment  (see  experiment  1  for  description).  In  this  experiment,  we  also  used  the 
GeneraaTs  3-D  audio  capability.  Figure  14  shows  the  robot  and  stereo  vision  and  stereo  audio 
features. 


Figure  14.  Unmanned  vehicle  “Generaal”  stereo  vision  and  3-D  audio  features. 

In  figure  14,  the  left  panel  shows  the  vehicle  with  a  forwardly  directed  red-tipped  pointer  at  the 
mid-front  for  approaching  targets,  vertical  antennas  for  visual  assistance  in  close  maneuvering, 
and  a  sensor  unit  on  a  pan-tilt-roll  motion  platfonn  with  3-D  audio  and  stereo  visual  sensors. 

The  sensor  unit  is  enlarged  in  the  upper-right  panel,  in  the  microphone  array  placed  in  their  3-D 
audio  position,  at  either  side  of  the  stereo  cameras.  The  lower-right  panel  shows  how  the  two 
microphone  arrays  were  placed  in  the  middle  position  right  above  the  stereo  cameras,  to  get  a 
directional  mono  sound. 

The  TNO  Generaal  is  a  manually  controlled  UGV  with  a  fast  and  powerful  pan-tilt-roll  camera 
or  sensor  system  that  can  accurately  mimic  human  head  movements  (Jansen  et  ah,  2010).  On  top 
are  two  cameras  for  providing  stereo  vision  at  the  control  station  and  two  microphone  arrays  that 
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can  be  positioned  at  either  side  for  spatial  3-D  audio  or  next  to  each  other  in  front,  thereby 
functioning  as  a  mono  audio  condition.  The  horizontally  positioned,  forwardly  directed  pointer 
in  front  of  the  vehicle  was  the  reference  point  for  the  participants  in  approaching  the  target  as 
closely  as  possible. 

3. 1.3. 2  The  Generaal  Telepresence  Controller.  The  same  Generaal  control  unit  used  during 
experiment  1  was  used  in  this  experiment.  However,  3-D  audio  reception  was  also  included. 

The  text  displays  used  in  experiment  1  were  not  necessary  in  this  experiment.  Figure  15  shows 
the  controller;  the  dual  image  is  shown  on  the  laptop.  The  user  sees  the  integrated  stereo  view  of 
the  laptop  image. 


Figure  15.  Telepresence  controller  (image  displayed  on  laptop). 


3. 1.3. 3  Building  Reconnaissance  Course.  The  building  reconnaissance  course  (figure  16)  was 
located  at  ARL’s  Human  Research  and  Engineering  Directorate  facility  at  the  McKenna  MOUT 
site,  Fort  Benning,  GA.  One  building  with  several  rooms  was  further  segmented  using  padded 
partitions.  Rooms  in  the  building  had  tables,  chairs,  computers,  and  other  furnishings. 
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Figure  16.  Indoor  route  reconnaissance  course. 


Figure  17  shows  the  general  layout,  with  positioning  of  target  items  for  the  audio  search  task.  It 
provides  a  map  of  the  target  environment  depicting  the  robot’s  starting  location,  the  decoy  and 
practice  targets  (gray  boxes),  and  real  targets  (F  and  I  were  used  in  practice  trials).  Targets  K 
and  J  are  in  a  room  adjacent  to  the  larger  room  with  the  other  targets.  The  gray  T- shapes 
subdivided  the  larger  room.  Figure  18  shows  a  panoramic  view  in  which  some  targets  are 
marked  by  labels  and  arrows. 

Eleven  possible  target  objects  varying  in  size  were  positioned  at  different  height  levels  in  the 
reconnaissance  environment.  The  underlined  items  were  used  as  targets  in  the  experimental 
trials;  the  italic  items  were  used  in  the  practice  trials;  and  items  A,  C,  and  K  were  decoy  targets 
that  were  never  used.  Wires  and  speakers  were  included  for  these  three  as  well. 

(A)  Soda  can  bomb  on  a  table 

(B) Hand  grenade  on  the  ground 

(C)  Soda  can  bomb  on  the  ground 

(D) Hand  grenade  near  the  ceiling 

(E)  Semtex  on  the  ground 

(F)  Bomb  shell  on  a  table 

(G) Pipe  bomb  on  the  ground 

(H)  Semtex  with  timer  on  a  chair 

(I)  Mine  on  a  water  container 

(J)  Land  mine  on  a  high  cupboard  shelf 

(K) Land  mine  on  a  high  cupboard  shelf 
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Figure  17.  Room  configuration  and  locations  of  audio  target  search  items. 


Figure  18.  Panoramic  view  on  the  target  environment. 
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The  control  station  was  located  in  a  tent  next  to  the  building  of  the  reconnaissance  environment 
(figure  19).  The  control  station  consisted  of  a  user  interface  with  an  NVIS  nVISOR  HMD 
(stereo  or  mono),  an  Xsens  MTi  motion  sensor  as  a  head  tracker,  stereo  headphones,  and  a 
Logitech  Dual  Action  game  controller. 


Figure  19.  Reconnaissance  environment  and  tent  with  control  station. 

3.1.3  Procedures 

3. 1.4.1  Soldier  Orientation.  The  Soldier  participants  reported  in  groups  of  four  for  1  day  each, 
from  0800  to  1800  daily.  After  arrival,  they  received  a  roster  number  to  identify  them 
throughout  the  evaluation  and  protect  privacy  information  on  the  data  sheets.  The  Soldiers  were 
oriented  on  the  purpose  of  the  study  and  their  participation.  They  were  briefed  on  the  objectives, 
procedures,  equipment,  the  reconnaissance  course,  and  reconnaissance  tasks.  Experimenters 
answered  any  questions  the  Soldiers  had  concerning  the  experiment.  They  were  also  told  how 
the  results  would  be  used  and  the  benefits  the  military  could  expect  from  the  investigation.  For 
this  experiment,  all  Soldiers  volunteered  to  participate  and  signed  consent  forms  (appendix  C). 

3. 1.4. 2  Demographics.  Soldiers  were  requested  to  provide  demographic  data  about  their 
physical  characteristics  and  military  experience,  including  their  knowledge  of  operating  remote- 
controlled  vehicles.  The  demographic  sheet  and  data  summary  are  provided  in  appendix  A. 

3. 1.4. 3  Training  and  Task  Demands.  No  specialized  experience  was  required  from  the  requested 
Soldiers.  A  representative  from  ARL  presented  an  overview  of  the  experiment,  the 
reconnaissance  courses,  and  task  demands.  Representatives  from  TNO  trained  the  Soldiers  on 
the  use  of  their  robot  control  systems.  Soldiers  were  given  hands-on  training  on  the  TNO 
controller  immediately  before  using  the  interface. 
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The  detailed  instructions  entailed  a  directed  walk  through  the  task  environment,  during  which  the 
TNO  researcher  pointed  out  the  1 1  locations  and  corresponding  items  that  could  be  used  as 
targets  (only  6  of  those  were  used  as  real  targets,  2  were  used  for  practice,  and  3  were  decoys). 
They  were  also  introduced  to  the  robot  and  its  sensor  system  and  trained  on  how  to  operate  it 
using  the  joystick  control.  Then  they  practiced  maneuvering  the  robot  (turn  around,  make  a 
90°  right  turn,  go  through  a  doorway,  make  a  turn  in  that  room,  and  return)  and  sensors  (look 
left,  right,  up,  and  down)  while  having  a  direct  view  of  the  robot.  The  participants  were  shown 
the  extensions  at  the  lateral  front  side  of  the  robot  that  assist  in  maneuvering  the  robot.  They 
were  instructed  to  make  sure  that  the  robot  did  not  collide  with  any  object,  furniture,  wall,  etc. 
The  second  part  of  the  instruction  was  at  the  control  station.  The  three  user  interface  setups  were 
explained.  For  each  setup,  the  participant  performed  a  practice  trial  to  a  practice  target.  This 
was  done  in  the  same  order  of  the  three  setups  as  used  in  the  experimental  trials,  with  F,  I  and 
again  F  as  practice  targets,  respectively. 

Before  starting  the  experimental  task,  the  participant  was  informed  about  the  possible  occurrence 
of  motion  sickness  when  using  HMDs.  The  experimenter  explained  the  first  symptoms  and 
explicitly  asked  the  participant  to  take  a  break  if  these  symptoms  were  experienced.  If  these 
symptoms  become  more  severe  after  continuation,  the  researchers  would  stop  this  Soldier’s 
participation  in  the  study. 

For  the  SA  task,  the  Soldiers  did  not  receive  further  training  on  the  controllers  because  the  SA 
tasks  were  performed  after  all  audio  search  tasks  were  completed.  For  the  SA  task,  Soldiers 
were  not  walked  through  the  rooms  but  were  told  there  would  be  many  target  items  and  to 
visually  search  for  as  many  as  possible,  trying  to  remember  each  item  and  its  location.  Training 
evaluation  questions  were  included  in  the  postiteration  questionnaire  regarding  adequacy  of 
training  and  requests  for  comments  and  suggestions.  Soldiers  indicated  high  satisfaction  with 
training.  A  seven-point  scale  from  extremely  bad  to  extremely  good  showed  the  mean  rating  for 
verbal  training  was  6.28  and  for  the  practice  session,  6.17.  Tasks  related  to  head-mounted 
camera  control  were  reported  to  be  easiest  to  learn.  Tasks  related  to  joystick  control  of 
movement  were  reported  as  more  difficult. 

3. 1.4.4  Task  Execution.  After  Soldiers  completed  training  on  a  particular  interface,  they 
completed  the  audio  search  task  with  each  controller  condition  in  counterbalanced  order.  At  the 
beginning  of  each  trial,  the  robot  was  placed  at  the  starting  position.  The  trial  started  with  the 
presentation  of  a  repetitive  sound  from  one  of  the  target  locations  (“help  .  .  .  help,”  every 
second).  The  participant  was  instructed  to  locate  where  the  sound  was  coming  from  by 
identifying  the  object  that  was  positioned  next  to  the  speaker  making  the  sound.  Once  identified, 
the  participant  had  to  maneuver  the  robot  to  approach  the  object  as  closely  as  possible  by 
minimizing  the  distance  between  the  object  and  a  pointer  attached  at  the  front  of  the  robot.  The 
maximum  allotted  time  was  set  at  10  min  for  each  trial.  If  a  participant  would  have  needed  more 
than  these  10  min  for  more  than  four  trials,  he  or  she  would  be  excluded  from  further 
participation. 
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The  participant  was  told  these  objects  were  considered  IEDs,  therefore  it  was  of  the  utmost 
importance  that  they  correctly  identify  the  target  as  fast  as  possible.  Corrections  were  allowed; 
the  final  identification  was  used  for  performance  measurements.  After  identification,  the 
participants  were  told  they  should  immediately  continue  to  approach  the  target.  During  this 
approach  it  was  most  important  not  to  collide  with  the  target  object.  They  should  approach  the 
object  as  closely  as  possible,  and  they  were  informed  that  we  would  use  time  to  final  approach  as 
a  performance  measure. 

During  task  execution,  a  TNO  experimenter  and  an  ARL  experimenter  observed  Soldier 
performance,  recorded  performance  measures,  and  noted  any  issues. 


After  all  Soldiers  completed  the  audio  search  task,  they  rotated  through  the  SA  task.  Twenty- 
two  items  were  placed  throughout  the  rooms.  Operators  were  requested  to  search  the  rooms, 
taking  note  of  all  items  of  interest,  and  to  try  to  remember  all  items  and  their  locations.  They 
were  then  given  a  diagram  that  specified  the  various  locations  where  the  items  were  placed, 
provided  with  a  pictorial  list  of  the  items  (figure  20),  and  asked  to  indicate  which  items  were 
clearly  remembered  to  be  at  particular  locations. 


Figure  20.  SA  search  task  items. 
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A  data  collector  accompanied  the  robot  to  report  start  and  stop  times,  times  off  course,  and 
driving  errors.  Another  data  collector  was  present  at  the  control  station  to  record  the  number  of 
objects  found  and  report  performance  measures.  Upon  completing  the  iteration,  the  Soldiers 
moved  to  a  questionnaire  area  to  fill  out  questionnaires  concerning  the  exercise  just  completed 
(appendix  A)  and  to  fill  out  a  NASA-TLX  concerning  the  level  of  workload  experienced 
(appendix  B). 

Once  Soldiers  completed  the  audio  search  task,  including  the  questionnaires,  they  joined  the 
other  Soldiers  in  a  waiting  area.  Soldiers  were  requested,  emphatically,  to  refrain  from 
discussing  their  experience  among  themselves.  Experimenters  were  nearby  to  monitor  and 
enforce  this  request.  Soldiers  remained  in  the  waiting  area  until  they  were  called  for  their  SA 
task.  After  both  tasks  were  completed,  Soldiers  filled  out  an  end-of-experiment  questionnaire. 
For  the  Soldiers,  participation  in  this  experiment  took  ~2  h  to  complete  both  iterations,  with  all 
data  collection  accomplished  in  ~8  h  in  a  single  day.  All  Soldiers  were  available  during  the  8  h. 

3.1.4  Experiment  Design 

This  experiment  was  a  within-subjects  design  with  three  levels  of  robot  control  interface  for  the 
audio  search  task: 

•  MJ  (Mono  Joystick):  Mono  audio  and  video  on  HMD,  with  joystick  control  for  robot 
movements  and  heading  of  camera  sensor  system.  Operators  must  control  the  camera  with 
the  joystick;  in  addition,  they  were  asked  not  to  move  their  heads. 

•  MH  (Mono  Head  Tracking):  Mono  audio  and  video  on  HMD,  with  joystick  control  for 
robot  movements  and  head  tracking  for  directing  the  sensor  system. 

•  T  (Telepresence):  Stereo  audio  and  video  on  HMD,  with  joystick  control  for  robot 
movements  and  head  tracking  for  directing  the  sensor  system. 

Each  participant  performed  the  sound  detection  task  18  times.  We  did  not  include  the  first  four 
participants  because  their  training  was  somewhat  different.  Participant  14  dropped  because  of 
nausea.  Each  of  the  six  targets  was  used  for  each  of  the  three  conditions.  After  each  trial,  the 
participant  switched  to  one  of  the  other  two  experimental  conditions.  For  example,  participant  5 
started  with  MH  in  the  first  trial  to  detect  target  B,  switched  to  MJ  in  the  second  trial  to  detect 
target  D,  then  to  T  for  target  H,  then  continued  with  MH  for  the  fourth  trial  for  localization  of 
target  G,  etc;  in  this  example,  the  pattern  of  MH-MJ-T  repeated  for  all  18  trials.  The  pattern  was 
counterbalanced  between  participants  (table  11).  Participant  22  had  to  stop  because  of  motion 
sickness. 


37 


Table  11.  Assignment  of  participants  to  repeating  pattern  of  user  interface 
setups. 


Participant  No. 

Order  of  Condition  Presentations 

10.  15.  20 

MJ  -  MH  -  T 

11,  16.21 

MJ  -  T  -  MH 

5.  8,  12 

MH  -  MJ  -  T 

7,  18 

MH  -  T  -  MJ 

6.  13,  19 

T  -  MJ  -  MH 

9,  17,  (22) 

T  -  MH  -  MJ 

The  order  of  targets  was  the  same  for  each  participant:  B,  D,  H,  G,  B,  E,  H,  J,  D,  J,  E,  B,  D,  H, 
G,  E,  G,  J.  This  target  order  guaranteed  that  all  targets  were  used  a  single  time  for  all  three  user 
interface  setups.  We  controlled  for  practice  effects  and  for  confounding  effects  for  combinations 
of  target  and  user  interface  by  varying  the  order  in  which  user  interface  setups  were  presented 
among  participants  while  keeping  the  order  of  targets  the  same  for  all  participants.  Table  12 
shows  the  order  of  trials  for  participant  5  as  an  example. 


Table  12.  Representative  target  presentation  for  operator  5. 


Trial 

User  Interface  Setup 

Target  ID 

Practice  1 

MH 

F 

Practice  2 

MJ 

I 

Practice  3 

T 

F 

1 

MH 

B 

2 

MJ 

D 

3 

T 

H 

4 

MH 

G 

5 

MJ 

B 

6 

T 

E 

7 

MH 

H 

8 

MJ 

J 

9 

T 

D 

10 

MH 

J 

11 

MJ 

E 

12 

T 

B 

13 

MH 

D 

14 

MJ 

H 

15 

T 

G 

16 

MH 

E 

17 

MJ 

G 

18 

T 

J 

For  the  SA  search  task,  there  were  only  two  controller  conditions:  full  telepresence  and  baseline, 
presented  in  a  within-subjects  experiment  design.  Each  Soldier  performed  the  SA  search  task 
with  each  controller  condition,  in  counterbalanced  order.  Soldiers  performed  in  pairs,  such  that 
when  Soldier  1  completed  his/her  first  condition,  they  then  proceeded  to  the  questionnaire  station 
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to  provide  feedback,  while  Soldier  2  begins  his/her  first  condition.  Soldier  1  then  returns  to 
complete  the  second  condition,  and  so  on. 

3.1.5  Measures 

Measures  included  aspects  of  performance,  workload,  and  Soldier  evaluations. 

3. 1.5.1  Performance  Measures.  Performance  measures  included  the  following: 

•  Audio  search  course  completion  time 

•  Time  to  approach  each  target  item 

•  Distance  to  each  target  item 

•  The  type  and  number  of  correct  objects  found  for  the  audio  search  task 

•  The  type  and  number  of  correct  objects  and  locations  for  the  SA  task 

3. 1.5. 2  Workload.  The  NASA-TLX  requires  the  rater  to  examine  multiple  combinations  of 
dimensional  levels  and  derive  an  overall  workload  score  (Hart  and  Staveland,  1988).  In  addition, 
the  Soldier  evaluations  included  ratings  of  difficulty  for  specific  tasks.  A  further  description  of 
the  TLX  is  included  in  appendix  B,  and  the  Soldier  evaluations  are  included  in  appendix  A. 

3. 1.5. 3  Situation  Awareness.  After  the  SA  task,  Soldiers  completed  a  questionnaire  that 
provided  locations  of  all  items  in  the  building.  They  were  also  provided  with  a  picture  of  each 
type  of  target  item  (all  were  different),  each  indicated  by  a  letter.  Each  Soldier  was  asked  to 
indicate  which  items  were  placed  in  each  location.  They  were  asked  not  to  guess  but  to  list  only 
the  items  they  were  fairly  certain  about. 

3. 1.5.4  Soldier-Based  Evaluations.  Soldiers  completed  a  postiteration  questionnaire  after  each 
task,  and  one  final  questionnaire.  Questionnaires  inquired  about  the  operator’s  experience  with 
the  controller  units,  overall  assessments,  and  comments.  The  questionnaires  were  designed  to 
elicit  Soldiers’  opinions  about  their  performance  and  experiences  with  each  of  the  control 
systems.  The  questionnaires  asked  the  Soldiers  to  rate  the  devices  on  a  seven-point  semantic 
differential  scale  ranging  from  “extremely  good/easy”  to  “extremely  bad/difficult.”  Copies  of 
each  questionnaire  are  included  in  appendix  A. 

3.1.6  Data  Analysis 

Descriptive  statistics  were  calculated  for  each  measure.  Performance  data  were  analyzed  using 
repeated  measures  ANOVA.  Follow-on  pairwise  comparisons  were  calculated  using  Holm’s 
Bonferroni  procedure  (Holm,  1979)  to  control  for  family- wise  error  rates.  Partial  eta  squared 
(q2p),  an  index  of  effect  size,  was  computed  for  each  ANOVA  (see  footnote  on  p.  15). 
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3.2  Results 


3.2.1  Audio  Search  Task 


3.2.2  Time  to  Find  Target  Item 


Figure  21  provides  the  mean  times  and  standard  deviations  for  time  to  approach  target  items  for 
each  controller  condition.  Time  was  shortest  for  full  telepresence  and  longest  for  the  mono 
joystick  condition.  Two-way  repeated  measures  ANOVA  of  both  display  type  and  target  type 
showed  significant  differences  among  the  means  for  display  (F(2,14)  =  12.42,  p  =  0.00,  rfp  = 
0.64)  and  for  targets  (F(5,ll)  =  15.14,  p  =  00,  rfp  =  0.873).  In  addition,  the  interaction  between 
display  type  and  targets,  while  not  significant  using  the  p-level  (due  to  df),  had  a  very  high  effect 
size,  indicating  that  displays  were  particularly  effective  for  some  targets  (F(10,6)  =  2.43,  p  =.14, 
rfp=  .80).  Follow-on  Holm’s  Bonferroni  paired  comparison  tests  for  audio  search  times  are 
provided  in  table  13. 


Audio  Search  Time  by  Display 
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80 


Audio  Search  Time 
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Telepresen 
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Audio  Search  Time 
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88.77 

113.5 

Std.  Deviation 

22.75 

36.13 

63.45 

Figure  21.  Mean  time  and  standard  deviations  for  time  to  approach  for  each  controller  condition. 


Table  13.  Holm’s  Bonferroni  paired  comparisons  for  mean  times  (audio  search  task). 


Pair 

t 

df 

Obtained  p 

Required  p 

Telepresence  vs.  mono  joystick 

4.345 

15 

c3 

o 

o 

d 

0.05 

Telepresence  vs.  mono  head  track 

4.232 

15 

o.oor 

0.0167 

Mono  joystick  vs.  mono  head  track 

2.337 

15 

0.034 

0.025 

J  Met  requirements  for  statistical  significance. 
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Figure  22  provides  the  mean  audio  search  time  by  display  and  target. 
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Figure  22.  Mean  audio  search  time  by  display  and  target. 

3.2.3  Time  to  Approach  Target  Item 

A  repeated  measure  ANOVA  on  time  to  target  identification  included  both  display  and  target  as 
variables  in  order  to  assess  degree  of  interaction.  It  indicates  a  main  effect  for  display  (F(2,14) 
=  15. 18, p  =.000,  rfp=  0.68)  and  a  main  effect  for  target  (F(5,ll)=  10.07, p  =.001,  rfp=  0.82). 
While  the  interaction  term  p- value  was  not  significant,  the  effect  size  was  very  high  (F(10,6)  = 
1.66;  p  =  0.276,  r|  p=  0.73).  Effects  can  be  seen  in  figure  23.  Post-hoc  Holm’s  Bonferroni 
results  are  provided  in  table  14.  All  comparisons  were  significant. 
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Figure  23.  Mean  time  to  approach. 


Table  14.  Holm’s  Bonferroni  paired  comparisons  for  mean  time  to  approach  (audio  search  task). 


Pair 

t 

df 

Obtained  p 

Required  p 

Telepresence  vs.  mono  joystick 

5.159 

15 

o.ooo11 

0.05 

Telepresence  vs.  mono  head  track 

3.387 

15 

0.004“ 

0.0167 

Mono  joystick  vs.  mono  head  track 

2.932 

15 

0.010“ 

0.025 

“Met  requirements  for  statistical  significance. 


3.2.4  Percentage  of  Correct  Target  Identifications 

Table  15  shows  the  percentage  of  targets  correctly  identified  by  display.  Two-way  ANOVA 
shows  that  the  differences  due  to  display  conditions  are  significant  (F(2,14)  =  9.5 l,p  =  0.002, 
rfp=  0.58).  In  addition,  differences  due  to  target  are  significant  (F(5,ll)  =  4.104,  p  =  0.024,  rfp 
=  0.651),  and  while  the  interaction  term  was  not  significant,  the  effect  size  was  very  high 
(F(10,6)  =  3.30,  p  =  0.08,  rfp  =  0.846).  Post-hoc  Holm’s  Bonferroni  indicate  that  the 
telepresence  was  significantly  higher  in  percentage  of  correct  identifications. 

Figure  24  illustrates  the  interaction.  Telepresence  was  generally  associated  with  higher 
percentages  but  was  particularly  helpful  for  some  targets. 
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Table  15.  Percent  correct  identifications  by  display  and  target. 


Pair 

t 

df 

Obtained  p 

Required  p 

Telepresence  vs.  mono  joystick 

3.496 

15 

0.003a 

0.05 

Telepresence  vs.  mono  head  track 

3.873 

15 

0.002“ 

0.0167 

Mon-joystick  vs.  mono  head  track 

0.689 

15 

0.502 

0.025 

“Met  requirements  for  statistical  significance. 
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Figure  24.  Percentage  of  targets  correctly  identified  by  display  and  target. 

3.2.5  Distance  to  Target  Item 

Data  collectors  observed  and  measured  the  distance  from  the  robot  camera  to  the  target  after  the 
target  was  approached.  Mean  distances  for  each  condition  are  listed  in  figure  25.  Differences 
due  to  display  type  were  not  significant  (F(2,13)  =  0.258,  p  =  0.776,  rfp=  0.038).  Differences 
due  to  target  type  were  significant  (F(5,10)  =  52.84,  p  =  0.00,  rfp  =  0.96).  The  interaction  term 
was  not  significant;  however,  the  effect  size  was  very  large  (F(10,5)  =  2.441,  p  =  0.168,  rfP  = 
0.83).  Means  and  standard  deviations  are  listing  in  appendix  A. 
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Figure  25.  Mean  distance  to  targets  by  display  condition  and  target. 

3.2.6  Mean  Driving  Errors 

A  repeated  measures  ANOVA  of  display  and  target  showed  a  significant  difference  due  to 
display  (F(2,13)  =  5.14,  p  =  0.02,  rfp  =  0.44)  but  not  for  targets  (F(5,10)  =  2.36,  p  =  0.12,  r|2p  = 
0.54)  or  their  interaction  (F(10.5)  =  1 .67,  p  =  0.29,  rfp  =  77).  However,  effect  sizes  are  large 
(see  figure  26).  Means  and  standard  deviations  are  listed  in  appendix  A. 
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Mean  Driving  Errors  by  Display  and  Target 


///  /// 
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Figure  26.  Mean  driving  errors  by  display  and  target. 

3.2.7  Spatial  Ability  Assessment 

Spatial  ability  was  assessed  through  the  Cube  Comparisons  Test*  (Ekstrom  et  al.,  1976).  This 
test  measures  the  ability  to  mentally  rotate  a  line  drawing  of  a  3-D  cube.  Soldiers  were  allotted  3 
min  to  mentally  rotate  and  respond  to  21  test  items.  The  dependent  measure  is  the  correct 
identification  of  the  mental  rotation  of  each  test  item  from  a  series  of  forced-choice  line 
drawings.  Soldiers  read  the  instructions  and  performed  sample  items  prior  to  the  test.  They  were 
encouraged  not  to  guess,  as  the  final  score  is  calculated  by  subtracting  the  number  wrong  from 
the  number  correct.  Scores  ranged  from  0  to  19,  with  a  mean  of  7.22  (sd  4.79). 

Spatial  ability  correlated  significantly  with  audio  search  measures.  For  telepresence  audio  search 
times,  spatial  score  (spa)  correlated  -.52  (p  =  0.02);  for  mono  joystick,  spa  correlated  -.65 
(p  =  0.00),  for  mono  head  tracking,  spa  correlated  -.65  (p  =  .00).  The  spa  was  entered  as  a 
covariate  in  repeated  measures  analyses  of  display  and  audio  search  time.  There  was  a 
significant  interaction  between  display  and  spa  (F  =  4.67,  p  =  .025),  as  reflected  in  the  different 
correlation  values  between  spa  and  audio  search  times  for  the  different  display  conditions.  The 
correlation  with  performance  was  lowest  for  telepresence,  with  the  implication  that  telepresence 
allowed  participants  with  lower  spa  to  perform  somewhat  better  than  the  other  conditions.  This 
can  be  seen  in  figure  27,  where  Soldiers  with  low  spa  performed  much  faster  in  the  telepresence 
condition  as  opposed  to  the  other  conditions. 


*Copyright  1962,  1976  by  Educational  Testing  Service. 
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Figure  27.  Mean  audio  search  times  by  spa  and  display. 

3.2.8  NASA-TLX  Workload  Ratings 

Soldiers  provided  direct  ratings  of  the  NASA-TLX  workload  scales.  Mean  ratings  are  provided 
in  figure  28.  Differences  were  significantly  different  for  mental  workload  (F(2,15)  =  20.98, 
p  =  0.00,  r|2p  =  0.74);  effort  (F(2,15)  =  9.44, p  =  0.00,  ri2p=  0.56),  and  frustration  (F(2,15)  = 

7.82,  p  =  0.01,  rfp=  0.51).  While  the  F  test  for  temporal  demand  was  not  significant  (F(2, 1 5) 

=  3.38,  p  =  0.06),  the  effect  size  was  moderately  high  (rfp  =  0.31).  Follow-on  Holm’s  Bonferroni 
comparisons  are  provided  in  table  16.  The  performance  subscale  is  not  a  measure  a  workload 
but  a  self-assessment  of  how  well  each  operator  rated  his  or  her  performance. 
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Figure  28.  Mean  NASA-TLX  workload  ratings  by  display. 


Table  16.  Paired  comparison  t-tests  with  Holm’s  Bonferroni  criteria. 


Mental 

t 

df 

Obtained  p 

Required  p 

Tele  -  mono  head  track 

-3.03 

16 

0.008“ 

0.05 

Tele  -  mono  joystick 

-6.22 

16 

0.000“ 

0.0125 

Mono  head  -  mono  joy 

-4.23 

16 

0.001“ 

0.025 

Effort 

— 

— 

— 

— 

Tele  -  mono  head  track 

-1.267 

16 

0.22 

0.05 

Tele  -  mono  joystick 

-3.85 

16 

0.001“ 

0.025 

Mono  head  -  mono  joy 

3.92 

16 

0.001“ 

0.0125 

Frustration 

— 

— 

— 

— 

Tele  -  mono  head  track 

-2.998 

16 

0.009“ 

0.05 

Tele  -  mono  joystick 

-4.079 

16 

0.001“ 

0.025 

Mono  head  -  mono  joy 

3.053 

16 

0.008“ 

0.0125 

p  <  .05,  two-tailed. 


3.2.9  Soldier  Feedback 

3.2.9. 1  Task  Difficulty  Ratings.  Soldier  responses  to  questionnaires  are  provided  in  full  in 
appendix  A.  Table  17  provides  mean  responses  to  items  regarding  task  difficulty  ratings  for  each 
display  condition.  Ratings  ranged  from  1  =  extremely  difficult  to  7  =  extremely  easy.  Soldier 
comments  expressed  higher  difficulty  with  the  mono  condition  to  locate  the  targets  and  higher 
difficulty  with  the  joystick  to  maneuver  the  robot.  This  is  also  reflected  in  the  ratings,  such  as 
the  ratings  for  capability  to  locate  target  objects  by  sound  location. 
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Table  17.  Mean  ratings  of  ease/difficulty  for  tasks. 


Task 

Mean  Response 

Telepresence 

Mono/Head 

Tracking 

Mono/Joystick 

Move  the  robot  in  the  desired  direction 

6.11 

5.74 

4.63 

Avoid  obstacles 

5.42 

5.00 

3.95 

Turn  around 

5.32 

4.95 

4.53 

Maneuver  around  corners 

5.42 

5.11 

4.47 

Back  up 

5.11 

4.74 

4.58 

Hear  the  target  sound  (sound  volume) 

5.47 

4.58 

4.11 

Locate  target  objects  by  sound  location 

5.47 

3.89 

3.26 

Locate  target  objects  by  sight 

6.00 

5.79 

4.89 

Find  targets  quickly 

5.42 

4.79 

3.89 

Know  your  location  in  the  room 

5.95 

5.89 

5.53 

Remember  target  locations 

5.63 

5.47 

5.21 

Move  quickly 

5.47 

4.84 

4.26 

Maneuver  probe  close  to  the  IED  object  (when 
requested) 

5.63 

5.32 

4.74 

Items  regarding  movement  of  the  robot  (items  1,  2,  3,  4,  5,  9,  12,  13)  were  averaged  to  create  a 
more  reliable  overall  assessment  of  robot  maneuverability.  Overall  means  ranged  from  4.38 
(mono  joystick),  to  5.05  (mono  head  track),  to  5.49  (telepresence).  The  overall  within-subjects 
F-test  was  significant  (F( 2,36)  =  16.68,  p  =  0.00,  r|2p=  0.48).  Paired  comparison  t-tests  with 
Holm’s  Bonferroni  criteria  are  listed  in  table  18.  All  paired  comparisons  met  the  criteria  for 
significance. 

Table  18.  Paired  comparison  t-tests  with  Holm’s  Bonferroni  criteria. 


Robot  Control:  Movement 

t 

df 

Obtained  p 

Required  p 

Tele  -  mono  head  track 

-3.04 

18 

0.007a 

.05 

Tele  -  mono  joystick 

-4.28 

18 

0.000a 

.0125 

Mono  head  -  mono  joy 

-4.26 

18 

0.000a 

.025 

dp  <  .05,  two-tailed. 


3. 2. 9. 2  Display/Control  Characteristics.  Soldier  responses  to  questionnaires  are  provided  in  full 
in  appendix  A.  Table  19  provides  mean  responses  to  items  regarding  display/control 
characteristics  for  each  display  condition.  Ratings  ranged  from  1  =  extremely  ineffective  to  7  = 
extremely  effective.  In  general,  ratings  were  higher  for  the  telepresence  controller  condition  and 
lowest  for  the  mono  joystick  condition.  A  repeated-measures  ANOVA  of  the  overall  rating  for 
each  controller  unit  (item  10)  resulted  in  F(2,34)  =  10.15,p  =  0.000,  rfp  =  0.37. 
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Table  19.  Mean  ratings  of  effectiveness  for  display  characteristics. 


Display/Control  Characteristics 

Mean  Response 

Telepresence 

Mono  Head 
Tracking 

Mono 

Joystick 

Resolution  (clarity)  of  the  display 

5.33 

5.28 

5.11 

Precision  of  camera  movement  control 

5.44 

5.22 

5.11 

Sensitivity  of  camera  movement  control 

5.17 

4.94 

4.67 

Time  lag  of  camera  movement  control 

4.83 

5.06 

4.88 

Controlling  camera  and  movement  at  the  same  time 

5.72 

5.67 

4.53 

Capability  to  hear  sound  (volume) 

5.50 

5.17 

4.83 

Capability  to  locate  sound  (direction) 

5.33 

4.00 

3.33 

Overall  ease  of  use 

5.94 

5.50 

4.72 

Adequacy  of  this  control  unit  for  tele -operating  a  robot 

5.67 

4.94 

4.61 

Overall  assessment  of  this  control  unit 

5.78 

5.00 

4.39 

Paired  comparison  (-tests  of  the  overall  assessment  item  are  listed  in  table  20.  Results  were 
significant  for  each  comparison. 

Table  20.  Paired  comparison  r-tests  with  Holm’s  Bonferroni  criteria. 


Overall  Assessment 

t 

df 

Obtained  p 

Required  p 

Tele  -  mono  head  track 

2.83 

17 

C3 

o 

© 

0.05 

Tele  -  mono  joystick 

3.65 

17 

o.oo“ 

0.0125 

Mono  head  -  Mono  joy 

2.37 

17 

0.03“ 

0.025 

a p  <  .05,  two-tailed. 


3. 2. 9. 3  Relevance  of  Capabilities  to  a  Reconnaissance  Mission.  Soldiers  were  also  asked  to  rate 
(1  =  extremely  bad  to  7  =  extremely  good)  to  what  extent  the  following  features  would 
contribute  to  a  recon  mission.  Results  are  listed  in  table  21.  The  3-D  audio,  stereo  vision,  and 
head-track  camera  all  received  high  mean  ratings. 

Table  21.  Mean  responses  to  relevance  of  each  feature  for  a 
reconnaissance  mission. 


Features 

Mean  Response 

3-D  audio 

6.56 

Stereo  vision 

6.22 

Head-tracked  camera 

6.06 

Mono  audio 

3.67 

Mono  vision 

4.11 

Joystick  camera 

4.56 

When  asked  which  controller  condition  was  preferred,  13  of  18  responses  were  in  favor  of  the 
telepresence  condition.  Reasons  provided  for  the  preference  are  included  in  table  22. 
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Table  22.  Consolidated  comments  regarding  display  characteristics. 


Soldier  Comments  Regarding  Telepresence 

No.  of  Responses 

Just  seemed  easier  to  use. 

2 

Easier  for  sight. 

1 

Easier  to  locate  target. 

1 

Firing  arm  is  free  for  use. 

1 

Natural. 

1 

The  most  impressive,  allowing  a  quicker  assessment  of  the  situation  and  greater  ease  of 
maneuver. 

1 

Easier  to  locate  and  identify  objects  while  being  able  to  move  your  head  (as  part  of  the 
camera/stereo  vision).  1  felt  the  3-D  audio  was  very  precise  and  so  was  the  head  tracking 
feature. 

1 

3-D  audio  helped  a  lot  for  target  identification  when  multiple  units  were  in  the  same  area. 

2 

3-D  audio  helped  to  locate  the  direction  of  the  sound  a  lot  faster. 

1 

Because  I  was  naturally  inclined  to  move  my  head  with  the  3-D  view.  With  video 
games,  I  was  used  to  the  joystick,  but  the  television  for  that  is  farther  away  and  less 
impressive;  with  this  system,  it  was  much  easier  to  use  the  head  tracking. 

1 

1  didn’t  notice  an  appreciable  difference  in  terms  of  the  stereo  vision  increasing  my  depth 
perception.  This  may  have  been  mainly  attributable  to  the  poor  quality  of  the  display. 

1 

Felt  the  most  precise.  I  could  move  quickly  and  fluidly  through  the  course. 

1 

Easiest  overall  control  of  box  and  easiest  ability  to  listen  to  where  the  sound  was  coming 
from. 

2 

The  head  tracking  was  extremely  helpful  in  the  overall  maneuvering  of  the  robot. 

1 

It  combined  the  head  movement  along  with  the  stereo  audio.  That  was  a  huge  plus. 

1 

3. 2. 9.4  Soldier  Discomfort.  Soldiers  indicated  whether  they  experienced  any  negative 
symptoms  from  the  controller  task.  Results,  listed  in  table  23,  indicate  that  several  symptoms 
were  experienced  by  several  Soldiers. 

Table  23.  Number  of  Soldiers  reporting  each  symptom  (telepresence). 


Symptoms 

No.  of  Responses 

Eyestrain 

8 

Tunnel  vision 

0 

Headaches 

8 

Motion  sickness 

5 

Nausea 

5 

Disorientation 

0 

Dizziness 

4 

Competition  between  eyes  for  vision  of  the  different 
scenes  at  which  they  are  looking 

4 

Other 

7 

Soldier  comments  regarding  the  symptoms  are  included  in  table  24. 
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Table  24.  Comments  regarding  symptoms. 


Soldier  Comments  Regarding  Telepresence 

No.  of  Responses 

Some  discomfort  from  weight. 

1 

The  headset  hurt  my  head  (too  heavy/hard/tight)  and  created  discomfort. 

4 

Fatigue. 

1 

Hands  started  to  tingle  with  the  increase  of  above  symptoms  (headaches,  motion 
sickness,  nausea,  dizziness). 

1 

I  was  only  able  to  do  tasks  in  groups  of  three  before  nausea  set  in.  After  each  break,  I 
was  able  to  resume,  but  the  nausea  came  back  at  a  faster  rate  each  time. 

1 

Headaches  may  be  due  to  seeing  double  during  the  breaks.  Also,  there  was  some  lag  at 
one  point. 

1 

Results  indicate  that  the  issues  of  eyestrain,  headache,  and  motion  sickness  persist,  as  does  need 
for  improvements  to  the  head-mounted  design  to  make  it  lighter.  Results  also  indicate  the  need 
for  further  improvements  to  resolution  and  reduced  time  lag  in  camera  control. 


4.  Overall  Discussion  and  Conclusions 


In  the  first  experiment,  we  compared  visual  telepresence  features  (e.g.,  head-controlled  camera, 
stereo  vision)  with  the  more  commonly  used  laptop  interface  for  robot  control.  This  first 
experiment  demonstrated  that  telepresence  capabilities  were  as  effective  as  current  laptop-based 
controllers  and  were  very  well  appreciated  by  users.  However,  results  suggested  effectiveness 
will  vary  because  of  task  demands.  In  addition,  Soldiers  provided  many  suggestions  for  design 
improvements. 

The  second  experiment  used  task  demands  more  fully  matched  to  telepresence  capabilities.  In 
the  second  investigation,  we  further  examined  the  utility  of  the  head  tracking  and  stereoscopic 
features  for  more  visually  demanding  indoor  reconnaissance  tasks.  We  also  investigated  the 
utility  of  3-D  (vs.  mono  audio)  in  audio  localization  tasks.  Robot  controller  tasks  included  visual 
navigation  with  audio  localization,  depth  perception,  and  overall  SA  perception.  Several 
telepresence  features  were  associated  with  higher  performance. 

4.1  Visual  Navigation  and  Audio  Localization 

For  this  task,  operators  were  given  a  tour  of  the  indoor  facility  and  training  on  robot  control  and 
navigation.  They  then  searched  the  rooms  while  the  controller  condition  varied  from  (a)  mono 
joystick  (mono  audio  and  mono  stereo,  with  joystick  control),  (b)  mono  head  track  (mono  audio 
and  mono  stereo  with  head  track  camera  control),  and  (c)  telepresence  (stereo  audio  and  stereo 
video  with  head  track  camera  control).  There  were  significant  differences  among  conditions 
with  regard  to  audio  search  time.  The  telepresence  condition  was  associated  with  the  fastest 
mean  time  (62.08  s),  which  was  significantly  faster  than  the  mono  head  track  condition  (82.70  s), 
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which  in  turn  was  significantly  faster  than  the  baseline  mono  joystick  condition  (105.56  s). 
Telepresence  was  also  associated  with  a  higher  mean  percentage  of  correct  identifications 
(80.7  %  compared  to  64.9%  and  57.8%)  with  differences  being  more  pronounced  for  some 
targets. 

In  addition,  mean  ratings  of  NASA-TLX  workload  for  task  1  were  significantly  lower  than  the 
mono  head  track  condition  for  mental  workload  and  frustration.  In  turn,  mean  ratings  of  mental 
workload  and  frustration  for  the  mono  head  track  condition  were  significantly  lower  compared  to 
the  baseline  mono  joystick  condition.  Mean  ratings  of  workload  effort  were  significantly  higher 
for  the  baseline  mono  joystick  condition  when  compared  to  the  mono  head  track  and 
telepresence  conditions.  Thus,  in  task  1,  the  telepresence  condition  was  associated  with  faster 
and  better  performance  and  lower  experience  of  workload.  The  head  track  condition  was 
associated  with  improved  performance  and  lower  workload  compared  to  the  mono  joystick.  The 
addition  of  stereo  audio  and  vision  further  improved  performance  and  lowered  workload  when 
added  to  the  head  track  capability. 

4.2  Depth  Perception 

Operators  were  asked  to  maneuver  the  robot  as  close  to  the  target  as  possible  without  touching  it. 
The  stereo  vision  feature  was  expected  to  assist  in  this  task;  however,  mean  differences  among 
the  display  conditions  were  quite  similar. 

4.3  Impact  of  Spatial  Ability 

Spa  was  found  to  be  a  direct  contributor  to  robot  control  performance  and  also  a  moderator  of  the 
effects  of  display  on  performance.  Spatial  ability  correlated  significantly  with  audio  search 
measures.  For  telepresence  audio  search  times,  spa  correlated  -0.52  (p  =  0.02);  for  mono 
joystick,  spa  correlated  -0.65  (p  =  0.00);  and  for  mono  head  tracking,  spa  correlated  -0.65 
ip  =  .00).  These  differences  among  the  correlations  were  significant.  The  spa  was  entered  as  a 
covariate  in  repeated  measures  analyses  of  display  and  audio  search  time.  There  was  a  significant 
interaction  between  display  and  spa  ( F  =  4.67,  p  =  .025),  which  is  reflected  in  the  different 
correlation  values  between  spa  and  audio  search  times  for  the  different  display  conditions.  The 
correlation  with  performance  was  lowest  for  telepresence,  with  the  implication  that  telepresence 
allowed  participants  with  lower  spa  to  perform  somewhat  better  than  did  the  other  conditions. 

4.4  Soldier  Discomfort 

While  refinements  to  telepresence  software  were  conducted  to  minimize  discomfort,  some 
Soldiers  reported  symptoms  of  eyestrain,  headaches,  motion  sickness,  nausea,  and  dizziness. 
These  symptoms  may  be  due  to  factors  related  to  stereo  vision  (e.g.,  competition  between  eyes, 
time  lag  in  camera  movements).  There  were  also  comments  with  regard  to  the  weight  and  fit  of 
the  headset. 
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4.5  Soldier  Feedback 

The  Soldiers  provided  feedback  through  structured  semantic  differential  rating  scales  (e.g.,  task 
difficulty,  ease  of  use,  comfort,  effectiveness)  and  verbal/written  comments.  For  ratings  of  task 
difficulty  and  SA,  the  telepresence  condition  was  rated  more  favorably  for  all  items  (13  out  of 
13).  Eight  of  the  items  referred  to  maneuvering  the  robot  (e.g.,  move  the  robot  in  the  desired 
direction,  avoid  obstacles,  turn  around)  and  were  averaged  to  create  a  more  reliable  measure  for 
analysis.  Overall  means  regarding  the  ease  of  robot  maneuvering  indicated  the  telepresence 
condition  was  associated  with  significantly  easier  robot  maneuvering  compared  to  the  mono  head 
track  condition,  which  in  turn  was  associated  with  significantly  higher  ratings  of  ease  than  the 
mono  joystick  condition. 

Soldiers  also  rated  characteristics  of  each  display  condition,  such  as  display  resolution,  precision, 
sensitivity,  etc.  In  general,  the  telepresence  condition  was  associated  with  the  highest  ratings  (9 
of  10),  with  the  exception  of  time  lag.  Telepresence  ratings  were  notably  higher  than  mono  head 
track  or  mono  joystick  for  capability  to  hear  sound  volume  (5.50  vs.  5.17  vs.  4.83),  capability  to 
locate  sound  (5.33  vs.  4.00  vs.  3.33),  overall  ease  of  use  (5.94  vs.  5.50  vs.  4.72),  adequacy  for 
teleoperation  (5.67  vs.  4.94  vs.  4.61),  and  overall  assessment  of  the  controller  (5.78  vs.  5.00  vs. 
4.39).  Paired  comparison  /-tests  of  the  overall  assessment  item  found  significant  differences  for 
each  condition  paired  comparison.  In  addition,  with  regard  to  contributions  to  an  operational 
reconnaissance  mission,  3-D  audio  was  rated  much  higher  than  mono  audio  (6.56  vs.  3.67), 
stereo  vision  was  rated  much  higher  than  mono  vision  (6.22  vs.  4.1 1),  and  the  head  track  camera 
was  rated  much  higher  than  the  joystick  (6.06  vs.  4.56). 

When  asked  which  controller  condition  was  preferred,  13  of  18  responses  were  in  favor  of  the 
telepresence  condition.  Reasons  provided  for  the  preference  included  overall  ease  of  use,  and  in 
particular,  ease  of  visual  search  and  target  localization.  Comments  included  descriptions  of  the 
telepresence  as  “intuitive,”  “easy,”  and  “second  nature.”  They  noted  that  the  telepresence 
allowed  the  hand/arm  to  be  free  and  a  quicker  assessment  of  the  situation.  The  3-D  audio 
combined  with  the  head  track  capability  was  particularly  valued  for  the  audio  localization  task. 
Soldiers  reported  they  could  not  localize  with  the  mono  audio,  and  that  robot  movement  and 
maneuvering  was  more  difficult  using  the  joystick. 

While  Soldiers  performed  well  with  both  controllers  having  telepresence  features,  they  also 
commented  on  how  a  telepresence  system  could  be  improved.  A  primary  issue  is  eyestrain  and 
motion  sickness.  Another  common  theme  is  that  of  physical  discomfort  from  the  weight  and 
bulk  of  the  head  track  equipment.  Some  Soldiers  reported  that  the  3-D  audio  could  be  further 
improved  in  that  it  was  difficult  to  tell  if  a  noise  was  coming  from  above  or  below.  Soldiers  also 
suggested  additional  capabilities,  such  as  a  rear  vision  camera,  a  more  effectively  placed  probe, 
crash/collision  sensors,  greater  field  of  view,  and  camera  zoom  capabilities.  They  also  noted  that 
the  system  would  have  to  be  integrated  with  a  combat  helmet  to  be  effective  in  operations. 
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In  summary,  the  telepresence  capabilities  are  promising,  but  further  improvements  are  indicated. 
Significant  differences  in  workload  and  performance  were  noted,  particularly  with  regard  to 
usefulness  of  the  remote  3-D  audio  capability.  Results  also  showed  the  benefit  of  head  tracking 
even  when  in  the  mono  audio  condition,  reflecting  the  importance  of  head  movement  for  sound 
localization,  as  found  elsewhere  (Toshima  and  Aoki,  2009).  However,  reports  of  eyestrain, 
headache,  or  motion  sickness  are  still  experienced  by  more  users  with  telepresence,  similar  to 
experiences  of  virtual  reality,  and  compounded  by  technical  issues,  such  as  camera  lag  or 
interocular  adjustments  regarding  the  stereo  display  (Kolasinski,  1995;  Van  Erp  et  al.,  2006). 

Soldiers  were  favorably  impressed  with  the  potential  that  current  systems  represent.  Further 
efforts  are  now  underway  to  address  these  improvements  and  investigate  each  aspect  of 
telepresence  with  regard  to  their  contributions  to  performance  in  different  task  demand  scenarios. 
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Appendix  A.  Detailed  Descriptive  Results 


This  appendix  appears  in  its  original  form,  without  editorial  change. 


A-l  -  Experiment  1:  Demographic  Results  (N  =  18) 


MOS 

RANK 

AGE 

DUTY  POSITION 

09S- 

15 

E4  -5 

25  years  (mean) 

OCS-  18 

11B  - 

1 

E5  -4 

14J- 

1 

OCS-9 

15A- 

1 

1 .  How  long  have  you  served  in  the  military?  _8_  months  (mean) 

2.  How  long  have  you  had  an  infantry-related  job?  3  months  (mean)  (N=l) 

3.  How  long  have  you  been  deployed  in  a  combat  area?  12  months  (mean)  (N=l) 

4.  With  which  hand  do  you  most  often  write?  17  Right  J_  Left 

5.  With  which  hand  do  you  most  often  fire  a  weapon?  17  Right  1  Left 

6.  a.  Do  you  wear  prescription  lenses?  10  No  _8_  Yes 

b.  If  so,  which  do  you  most  often  wear?  _5_  Glasses  _3_  Contacts 

c.  Which  is  your  dominant  eye?  12  Right  _4_  Left  _2_  NR 

d.  Do  you  have  any  vision  related  problem?  15  No  _3_  Yes 
If  so,  what?  Color  vision  deficiency,  astigmatism  (2) 

7.  a.  How  often  do  you  use  a  computer? 

0  Never  _0_  Infrequently  _2_  Sometimes  16  Daily 
b.  How  often  do  you  use  a  computer  on  a  weekly  basis? 

3  1-10  hrs  9_  1 1-20  hrs  _4_  21-30  hrs  _2_  More  than  30  hrs 

8.  Do  you  have  experience  with  any  type  of  digital  or  electronic  military  displays  (e.g.,  FBCB2, 
fire  control  systems  FB  (ITAS,  IBAS,  LW,  etc.)?  16  No  _2_  Yes 

9.  a.  Have  you  ever  used  a  robotic  system?  15  No  3_  Yes 

b.  If  so,  what  type?  PLC  (Programmable  Logics  Controller),  civilian  robots, 

c.  Please  describe  the  conditions  under  which  you  used  the  robotic  system. 
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Industrial/Manufacture.  1 

College/classroom  environment,  1 

We  built  and  controlled  our  own  robot  in  high  school  and  then  battled  each  1 
other. 

10.  Please  rate  your  skill  level  for  each  of  the  following  activities? 

None  Beginner  Intermediate  Expert 

12  3  4 


ACTIVITY 

MEAN  RESPONSE 

Operating  ground  unmanned  vehicles 

1.11 

Operating  aerial  vehicles 

1.28 

Target  detection  and  identification 

1.78 

Playing  commercial  video  games 

2.83 

Training  with  Army  video  simulations 

1.83 

Using  Microsoft  Office  products 

3.17 
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A-2  -  Experiment  1:  Performance  Data  for  Indoor  and  Outdoor  Reconnaissance  Tasks 


PERFORMANCE  DATA  from  Observers 


DETECTIONS.  In  this  set  of  data,  all  items  that  were  detected  were  counted,  whether  they  were 
correctly  identified  or  not. 


\  Recon  Items  :  Building  =  B  (DETECTIONS)  | 

Laptop 

Telepresence  j 

N 

Mean 

N 

Mean 

1 

Soda  Can  bomb  (diet  pepsi) 

8 

1.00 

8 

1.00 

2 

Soda  Can  bomb  (diet  coke) 

8 

1.00 

8 

1.00 

3 

Soda  Can  bomb  (diet  pepsi) 

8 

1.00 

8 

1.00 

4 

Land  mine  Grey 

8 

1.00 

8 

1.00 

5 

Land  mine  green  and  tan 

8 

1.00 

8 

1.00 

6 

Land  mine  tan 

8 

1.00 

8 

1.00 

7 

cardboard  box 

8 

1.00 

8 

1.00 

8 

picture  of  Statue  of  Liberty 

8 

.75 

8 

.50 

9 

suicide  bomber  vest 

8 

1.00 

8 

1.00 

10 

suicide  bomber  vest 

8 

1.00 

8 

1.00 

11 

suicide  bomber  vest 

8 

1.00 

8 

1.00 

12 

map  with  New  York  circled 

8 

.88 

8 

.75 

13 

briefcase  bomb 

8 

1.00 

8 

1.00 

14 

Timer 

8 

1.00 

8 

1.00 

15 

6v  battery 

8 

1.00 

8 

1.00 

16 

Wire 

8 

1.00 

8 

1.00 

17 

initiation  device 

8 

1.00 

8 

1.00 

18 

PDA 

8 

1.00 

8 

1.00 

19 

cell  phone 

8 

.88 

8 

1.00 

20 

pipe  bomb 

8 

1.00 

8 

1.00 

21 

82mm  mortar  round 

8 

1.00 

8 

1.00 

22 

122mm  projectile 

8 

1.00 

8 

1.00 

23 

130mm  rocket  round 

8 

1.00 

8 

1.00 

M_ALL  (total) 

8 

22.5000 

8 

22.25 

M_ALL  (%) 

8 

0.98 

8 

0.97 

Valid  N  (listwise) 

8 

8 
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!  Recon  Items  :  Building  =  C  (DETECTIONS)  ! 

Laptop 

Telepresence  ] 

N 

Mean 

N 

Mean 

1 

cardboard  box 

8 

1.00 

8 

1.00 

2 

PDA 

8 

1.00 

8 

1.00 

3 

Fan 

8 

1.00 

8 

1.00 

4 

AK47 

8 

1.00 

8 

1.00 

5 

AK47 

8 

1.00 

8 

1.00 

6 

AK47 

8 

1.00 

8 

1.00 

7 

rpg 

8 

1.00 

8 

1.00 

8 

12v  battery 

8 

.88 

8 

1.00 

9 

jumper  cables 

8 

1.00 

8 

1.00 

10 

chair 

8 

1.00 

8 

1.00 

11 

table 

8 

1.00 

8 

1.00 

12 

lamp 

8 

1.00 

8 

1.00 

13 

wire 

8 

.75 

8 

.63 

14 

rope 

8 

1.00 

8 

.88 

15 

wad  of  wire 

8 

.63 

8 

.13 

16 

ACU  uniform 

8 

1.00 

8 

1.00 

17 

ACU  uniform 

8 

1.00 

8 

1.00 

18 

bedding 

8 

1.00 

8 

1.00 

19 

bloody  rags 

8 

1.00 

8 

1.00 

20 

naked  mannequin 

8 

1.00 

8 

1.00 

21 

sandbag  over  head 

8 

.75 

8 

.88 

22 

chair 

8 

.88 

8 

.75 

23 

bedding 

8 

1.00 

8 

1.00 

M_ALL  (total)  8 

M_ALL  (%) 


21.8750 

0.95 


21.2500 

0.92 


IDENTIFICATIONS.  In  this  set  of  data,  only  items  that  were  correctly  identified  were  counted. 


i  Recon  Items  :  Building  =  B  (IDENTIFICATIONS)  1 

Laptop 

Telepresence  j 

N 

Mean 

N 

Mean 

1 

Soda  Can  bomb  (diet  pepsi) 

8 

1.00 

8 

1.00 

2 

Soda  Can  bomb  (diet  coke) 

8 

1.00 

8 

1.00 

3 

Soda  Can  bomb  (diet  pepsi) 

8 

1.00 

8 

1.00 

4 

Land  mine  Grey 

8 

.75 

8 

.63 

5 

Land  mine  green  and  tan 

8 

.25 

8 

.75 

6 

Land  mine  tan 

8 

.63 

8 

.63 

7 

cardboard  box 

8 

1.00 

8 

1.00 

8 

picture  of  Statue  of  Liberty 

8 

.75 

8 

.50 

9 

suicide  bomber  vest 

8 

1.00 

8 

1.00 

10 

suicide  bomber  vest 

8 

1.00 

8 

1.00 

11 

suicide  bomber  vest 

8 

1.00 

8 

1.00 

12 

map  with  New  York  circled 

8 

.88 

8 

.75 

13 

briefcase  bomb 

8 

1.00 

8 

1.00 

14 

Timer 

8 

1.00 

8 

1.00 

15 

6v  battery 

8 

1.00 

8 

.75 

16 

Wire 

8 

.88 

8 

.50 

17 

initiation  device 

8 

.50 

8 

.63 

18 

PDA 

8 

.63 

8 

.50 

19 

cell  phone 

8 

.88 

8 

.50 

20 

pipe  bomb 

8 

1.00 

8 

.88 

21 

82mm  mortar  round 

8 

.88 

8 

1.00 

22 

122mm  projectile 

8 

.88 

8 

1.00 

23 

130mm  rocket  round 

8 

1.00 

8 

1.00 

M_ALL  (total) 

19.87 

8 

19.0000 

M_ALL  (%) 

0.86 

8 

0.83 

Valid  N  (listwise) 

8 

8 

8 

1.00 
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1  Recon  Items  :  Building  =  C  (IDENTIFICATIONS)  | 

Laptop 

Telepresence  ] 

N 

Mean 

N 

Mean 

1 

cardboard  box 

8 

1.00 

8 

1.00 

2 

PDA 

8 

.88 

8 

1.00 

3 

Fan 

8 

1.00 

8 

1.00 

4 

AK47 

8 

1.00 

8 

1.00 

5 

AK47 

8 

1.00 

8 

1.00 

6 

AK47 

8 

1.00 

8 

1.00 

7 

rpg 

8 

.88 

8 

1.00 

8 

12v  battery 

8 

.88 

8 

1.00 

9 

jumper  cables 

8 

1.00 

8 

1.00 

10 

chair 

8 

1.00 

8 

1.00 

11 

table 

8 

1.00 

8 

1.00 

12 

lamp 

8 

1.00 

8 

1.00 

13 

wire 

8 

.63 

8 

.63 

14 

rope 

8 

1.00 

8 

.88 

15 

wad  of  wire 

8 

.63 

8 

.13 

16 

ACU  uniform 

8 

.88 

8 

1.00 

17 

ACU  uniform 

8 

1.00 

8 

1.00 

18 

bedding 

8 

1.00 

8 

1.00 

19 

bloody  rags 

8 

1.00 

8 

1.00 

20 

naked  mannequin 

8 

1.00 

8 

1.00 

21 

sandbag  over  head 

8 

.75 

8 

.88 

22 

chair 

8 

.88 

8 

.75 

23 

bedding 

8 

1.00 

8 

1.00 

M_ALL  (total) 

8 

21.3750 

8 

21.2500 

Performance  in  BUILDINGs:  Total  time.  Number  of  driving  errors 


N 

Minimum 

Maximum 

Mean 

Std. 

Deviation 

Laptop_Bldg_time 

16 

4:28:00 

15:00:00 

9:39:11.250 

2:49:30.032 

Laptop_Bldg_errors 

16 

0 

7 

2.31 

2.024 

Telepresence_Bldg_time 

16 

6:22:00 

12:45:00 

10:02:14.999 

1:32:10.490 

Telepresence  _Bldg_errors 

16 

0 

6 

2.38 

1.408 

Valid  N  (listwise) 

16 

Performance  OUTSIDE:  Time,  Driving  errors,  Waypoints,  Off-course,  False  detects 


N 

Minimum 

Maximum 

Mean 

Std. 

Deviation 

Laptop  _Outside_time 

16 

3:57:00 

14:07:00 

9:24:07.500 

3:24:11.678 

Laptop  _Outside_errors 

16 

0 

0 

.00 

.000 

Telepresence  _Outside_time 

16 

5:12:00 

14:30:00 

9:45:44.999 

2:57:37.290 

Telepresence  _Outside_errors 

16 

0 

0 

.00 

.000 

Laptop  _wpt_all 

16 

2.00 

3.00 

2.6875 

.47871 

Telepresence  _wpt_all 

16 

2.00 

3.00 

2.6250 

.5000 

Laptop  _#times  off  course 

16 

0 

0 

.00 

.000 

Telepresence  _#times  off  course 

16 

0 

1 

.06 

.250 

Laptop  _#  false  detects 

16 

0 

2 

.50 

.730 

Telepresence  _#  false  detects 

16 

0 

4 

.50 

1.033 

Laptop  _#  false  detects  corrected 

16 

0 

2 

.13 

.500 

Telepresence  _#  false  detects 
corrected 

16 

0 

1 

.06 

.250 

Valid  N  (listwise) 

16 
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OUTSIDE:  Targets,  %  detected,  mean  distance  of  detected  targets 


|  Laptop 

TELEPRESENCE  | 

N 

Mean 

Std. 

Deviation 

N 

Mean 

Std. 

Deviation 

Outside_l 

16 

.63 

.500 

16 

.25 

.447 

l_distance 

10 

16.60 

7.214 

4 

27.50 

8.660 

Outside_2 

16 

.56 

.512 

16 

.19 

.403 

2_distance 

9 

13.00 

5.292 

3 

14.67 

4.619 

Outside_3 

16 

.31 

.479 

16 

.44 

.512 

3_distance 

5 

6.60 

1.673 

7 

7.71 

3.498 

Outside_4 

16 

.44 

.512 

16 

.19 

.403 

4_distance 

7 

10.86 

3.532 

3 

11.00 

6.928 

Outside_5 

16 

.44 

.512 

16 

.19 

.403 

5_distance 

7 

7.00 

3.215 

3 

11.00 

4.583 

Outside_6 

16 

.75 

.447 

16 

.56 

.512 

6_distance 

12 

13.92 

6.127 

9 

14.33 

3.775 

Outside_7 

16 

.75 

.447 

16 

.75 

.447 

7_distance 

12 

20.42 

5.534 

12 

17.42 

4.680 

Outside_8 

16 

.37 

.500 

16 

.31 

.479 

8_distance 

6 

10.83 

4.401 

5 

11.80 

3.271 

Outside_9 

16 

.44 

.512 

16 

.75 

.447 

9_distance 

7 

11.71 

4.957 

12 

10.83 

2.443 

Outside_10 

16 

.69 

.479 

16 

.87 

.342 

10_distance 

11 

15.91 

9.279 

14 

17.93 

9.000 

Total  Targets 

16 

5.3750 

2.09364 

16 

4.5000 

1.63299 
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A-3  -  Experiment  1:  Post-Iteration  Questionnaire  Results 


1.  Using  the  scale  below,  please  rate  your  ability  to  perform  each  of  the  following  tasks  based  on 
your  experience  with  the  robot  controller  that  you  just  used. 

1  2  3  4  5  6  7 

Extremely  difficult  Very  Difficult  Difficult  Neutral  Easy  Very  easy  Extremely  easy 


TELEPRESENCE:  OUTDOORS 

N 

Mean 

Std.  Deviation 

Skewness 

Kurt  os  is 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Move  the  robot  in  the  correct 
direction 

16 

4.4375 

1.26326 

-.293 

.564 

-.872 

1.091 

Avoid  obstacles 

16 

4.3125 

1.19548 

.108 

.564 

-1.598 

1.091 

Avoid  potholes 

14 

4.2143 

1.12171 

.276 

.597 

-1.310 

1.154 

Identify  any  other  terrain  features 
that  makes  it  difficult  to 
maneuver  through  the  terrain 

16 

4.7500 

1.23828 

.060 

.564 

-.962 

1.091 

Anticipate  whether  the  ground 
clearance  of  the  vehicle  will 
allow  negotiation  of  rugged 
terrain 

15 

4.1333 

1.06010 

.116 

.580 

-1.557 

1.121 

Anticipate  whether  the  turn  radius 
of  the  vehicle  will  allow  a  turn 

16 

4.8750 

1.45488 

-.942 

.564 

.249 

1.091 

Identify  if  the  robot  is  on  the 
correct  path 

16 

5.1250 

1.82117 

-1.270 

.564 

.418 

1.091 

Navigate  far  enough  ahead  to 
plan  route  in  advance 

16 

5.5625 

1.26326 

-.614 

.564 

-.638 

1.091 

Navigate  well  enough  to  drive  at 
slowest  speeds 

15 

5.7333 

1.03280 

-.282 

.580 

-.917 

1.121 

Navigate  well  enough  to  drive  at 
medium  speeds 

14 

5.6429 

1.27745 

-.759 

.597 

-.286 

1.154 

Navigate  well  enough  to  drive  at 
fastest  speeds 

14 

5.0000 

1.66410 

-.818 

.597 

-.394 

1.154 

Finish  the  course  quickly 

16 

4.7500 

1.39044 

-.680 

.564 

.318 

1.091 

Find  IEDs  and  other  objects  of 
interest 

16 

4.2500 

1.61245 

-1.009 

.564 

.135 

1.091 

Navigate  to  next  waypoint 

16 

4.8125 

1.60078 

-.989 

.564 

.523 

1.091 

Take  pictures 

15 

5.9333 

1.66762 

-1.479 

.580 

1.059 

1.121 

Control  the  camera  (viewing 
direction) 

4 

5.2500 

1.70783 

-.753 

1.014 

.343 

2.619 

Make  robot  understand  you 

10 

5.2000 

1.54919 

-.188 

.687 

-1.276 

1.334 

Being  aware  of  the  direction  the 
camera  is  pointed  within  it  is  not 
centered  on  the  front  of  the  robot 

4 

4.7500 

1.25831 

-1.129 

1.014 

2.227 

2.619 

Move  the  camera  direction 

12 

6.0833 

1.44338 

-2.349 

.637 

6.264 

1.232 

Maintain  situation  awareness 

16 

5.2500 

1.65328 

-1.366 

.564 

1.726 

1.091 

Overall  ability  to  perform  this 
reconnaissance 

16 

4.9375 

1.28938 

-1.149 

.564 

.341 

1.091 
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TELEPRESENCE:  OUTDOORS 

N 

Mean 

Std.  Deviation 

Skewness 

Kurt  os  is 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Move  the  robot  and  camera  angle 
at  the  same  time 

12 

4.6667 

1.92275 

-.812 

.637 

-.664 

1.232 

Maintain  direction  to  way-point 
while  searching  for  items 

12 

5.1667 

1.46680 

-1.171 

.637 

.686 

1.232 

Laptop:  OUTDOORS 

N 

Mean 

Std.  Deviation 

Skewness 

Kurt  os  is 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Move  the  robot  in  the  correct 
direction 

16 

5.3125 

1.13835 

-.708 

.564 

.717 

1.091 

Avoid  obstacles 

15 

5.1333 

1.40746 

-.802 

.580 

.361 

1.121 

Avoid  potholes 

14 

5.0000 

1.70970 

-1.293 

.597 

1.516 

1.154 

Assess  down  slopes  for 
navigability 

6 

4.8333 

1.47196 

.418 

.845 

-.859 

1.741 

Assess  side  slopes  for 
navigability 

6 

4.8333 

1.47196 

.418 

.845 

-.859 

1.741 

Identify  any  other  terrain  features 
that  makes  it  difficult  to 
maneuver  through  the  terrain 

16 

5.6875 

.94648 

-.352 

.564 

-.471 

1.091 

Anticipate  whether  the  ground 
clearance  of  the  vehicle  will 
allow  negotiation  of  rugged 
terrain 

14 

4.1429 

1.35062 

-.736 

.597 

.890 

1.154 

Anticipate  whether  the  turn  radius 
of  the  vehicle  will  allow  a  turn 

15 

4.6667 

1.71825 

-.278 

.580 

-1.293 

1.121 

Identify  if  the  robot  is  on  the 
correct  path 

16 

5.5625 

1.45917 

-1.318 

.564 

1.295 

1.091 

Navigate  far  enough  ahead  to 
plan  route  in  advance 

16 

5.8125 

1.22304 

-.844 

.564 

.167 

1.091 

Navigate  well  enough  to  drive  at 
slowest  speeds 

15 

6.3333 

.81650 

-.740 

.580 

-1.022 

1.121 

Navigate  well  enough  to  drive  at 
medium  speeds 

15 

6.0667 

1.09978 

-1.635 

.580 

3.411 

1.121 

Navigate  well  enough  to  drive  at 
fastest  speeds 

14 

5.5714 

.93761 

-1.546 

.597 

3.852 

1.154 

Finish  the  course  quickly 

16 

5.1875 

1.04682 

-.422 

.564 

-.201 

1.091 

Find  IEDs  and  other  objects  of 
interest 

16 

4.9375 

1.65202 

-.698 

.564 

-.743 

1.091 

Navigate  to  the  next  waypoint 

16 

5.8750 

1.36015 

-1.198 

.564 

.647 

1.091 

Take  pictures 

16 

6.5625 

.81394 

-2.348 

.564 

6.262 

1.091 

Control  the  camera  (viewing 
direction) 

4 

5.2500 

1.25831 

1.129 

1.014 

2.227 

2.619 

Make  the  robot  understand  you 

10 

6.3000 

.94868 

-.742 

.687 

-1.640 

1.334 

Being  aware  of  the  direction  the 
camera  is  pointed  within  it  is  not 
centered  on  the  front  of  the  robot 

4 

4.7500 

1.89297 

-1.659 

1.014 

2.615 

2.619 

Move  the  camera  direction 

12 

5.2500 

1.48477 

-.312 

.637 

-1.270 

1.232 
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Laptop:  OUTDOORS 

N 

Mean 

Std.  Deviation 

Skewness 

Kurt  os  is 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Maintain  situation  awareness 

16 

5.2500 

1.18322 

-.276 

.564 

.361 

1.091 

Overall  ability  to  perform  this 
reconnaissance 

16 

5.6250 

.80623 

-.027 

.564 

-.130 

1.091 

Move  the  robot  and  the  camera 
angle  at  the  same  time 

11 

4.9091 

1.57826 

-.379 

.661 

-1.823 

1.279 

Maintain  direction  to  way-point 
while  searching  for  items 

12 

4.9167 

1.50504 

-.024 

.637 

-1.463 

1.232 

No.  of 

Comments  Responses 

TELEPRESENCE:  OUTDOORS 

Great  piece  of  equipment  while  in  the  field.  2 

Easily  maneuvered  and  easily  identified  objects.  2 

Much  easier  to  move  and  adjust  camera.  1 

Easier  and  more  efficient  than  the  Laptop.  1 

Moving  the  camera  and  robot  at  the  same  time  was  much  easier  with  this 
device.  1 

Poor  resolution  of  “mono”  led  to  some  misidentification.  Neverthe-less,  well 
completed.  Controls  very  sensitive  and  responsive,  and  had  no  trouble 
perfectly  substituting  robot’s  eyes  for  mine.  1 

I  wish  there  was  a  way  to  control  volume  on  the  handset.  It  was  a  bit  loud  at 
times,  especially  when  going  over  portions  of  dense  conglomerations  of  rocks.  1 

Slight  disorientation  while  head  movement  was  quickly  changed.  1 

Head  movements  were  most  difficult  to  function  while  moving  quickly.  1 

Began  to  get  nauseated  often  in  the  later  part  of  the  outdoor.  I  felt  as  if  there 
was  a  lag  between  my  motions  and  the  actual  motions  the  robot  was  making.  1 

Most  of  the  difficulties  I  had  were  more  with  the  robot,  but  the  camera  system 
at  times  jumped  around  and  it  could  be  a  little  confusing  as  to  where  the 
camera  is  pointing  in  relation  to  the  robot.  1 

Difficult  to  adjust  camera  direction  at  times.  1 

Sometimes  camera  detail  not  as  precise.  Camera  quality  also  not  as  good.  1 

Camera  is  a  lot  harder  to  keep  steady  and  control  with  the  robot.  1 

With  camera  sitting  so  high  on  the  robot,  it  was  difficult  for  me  to  watch  where 
the  robot  was  going,  and  if  it  was  staying  on  course  (going  where  I  thought  it 
was  going).  1 

I  had  trouble  navigating  the  inside  course  mainly  because  of  the  turn  going  into 
the  torture  room.  I  am  not  sure  if  the  headset  display  was  more  difficult  to  use 
around  the  corner,  but  I  did  think  it  was  easier  to  search  for  items  while 
moving  the  robot.  1 

The  robot  seemed  to  veer  when  I  moved  forward  and  also  looked  around.  Do 
the  wheels  return  to  zero  or  do  they  stay  angled  to  continue  turning?  It  threw 
me  off  during  Telepresence.  Overall,  however,  I  felt  as  though  the 
Telepresence  definitely  helped  to  identify  threats.  Navigation  was  a  bit  tricky,  1 
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No.  of 

Comments  Responses 

but  is  manageable  with  practice.  Also,  the  rough  terrain  made  movement 
difficult. 

The  terrain  was  difficult  on  the  steering.  1 

Telepresence  was  actually  worse  for  the  outside  reconnaissance.  This  is  only 
due  to  the  fact  that  the  video  resolution  of  the  Telepresence  headset  device 
display  is  terrible.  I  couldn’t  clearly  see  the  IEDs,  etc.,  although  I  have  perfect 
vision,  and  it  was  much  easier  to  identify  the  items  by  simply  watching  the 
video  on  the  monitor.  1 

To  navigate  to  the  waypoints  with  Telepresence,  I  had  to  keep  switching  back 
to  the  map,  and  drive  to  the  next  point.  I  can’t  see  IEDs  and  other  items  while 
I’m  driving  in  “map”  mode.  Why  not  just  have  a  small  map  at  the  bottom  of 
the  screen?  Then  I  can  see  where  I  am,  and  view  the  surroundings  at  the  same 
time.  1 

There  was  not  a  large  advantage  of  using  Telepresence  outside  because  there 
was  plenty  of  room  and  time  to  look  around.  The  difference  in 
maneuverability  and  situation  awareness  didn’t  become  clear  until  the  indoor 
reconnaissance.  1 

1st  problem:  The  alignment  of  me  facing  forward  with  the  robot  eyes  facing 
forward  with  the  robot  body  was  gradually  drifting  out  of  alignment.  This 
meant  that  after  some  time,  I  would  face  front,  but  the  robot  was  not  facing 
straight  ahead  but  off  to  the  side,  and  I  would  have  to  re-calibrate  the  robot  to 
face  front.  This  was  extremely  confusing  and  bothersome.  Front  should 
always  be  front.  When  I  face  straight  ahead,  I  want  my  robot  facing  straight 
ahead.  1 

2nd  problem:  There  were  repeated  errors  with  the  robot.  Little  worked  as  it 
should.  Robot  kept  spinning  in  circles  on  its  own.  Camera  would  freeze,  and 
computer  had  to  be  re -booted.  Commands  entered  (such  as  “view  satellite 
map”  button)  wouldn’t  work.  When  I  push  a  button  or  command  the  robot,  I 
need  it  to  carry  out  that  action  IMMEDIATELY,  and  WITHOUT  FAIL. 

Actually,  an  anomaly  can  be  overcome  as  long  as  the  robot  is  reacting 
consistently.  This  robot  was  not  consistent;  I  could  never  trust  it  to  move  or 
look  at  the  directions  I  was  trying  to  command  it.  1 

Suggestion:  use  a  head-mounted  sensor  to  direct  the  robot’s  eyes,  but  let  the 
user  view  the  video  (mono)  on  a  Laptop  screen.  Resolution  was  much  better 
on  the  screen,  “stereo  vision”  was  cool,  but  it  did  nothing  to  help  me  see.  As  a 
result,  I  operate  in  mono  the  whole  time  because  in  stereo  it  would  be  right  on 


top  of  an  item  and  still  not  see  it  until  I  switched  to  mono.  1 

Difficult  to  see  objects  on  ground  in  general,  let  alone  when  moving.  1 

Display  quality  made  it  hard  to  distinguish  what  an  object  was  at  more  than  6 
feet  out.  Objects  tended  to  blend  in  with  terrain.  1 

Laptop:  OUTDOORS 

Very  easy  to  use.  2 

With  practice,  this  style  could  be  mastered.  1 
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No.  of 

Comments  Responses 

The  movement  speed  of  the  robot  was  excellent.  I  tried  different  speeds  which 
turned  out  to  be  very  easy  to  handle.  1 

Navigation  outdoors  is  much  easier  when  a  map  is  present  to  orient  off  of, 
especially  when  the  direction  of  travel  is  clearly  marked.  1 

This  system  while  not  having  the  “cool”  factor  of  the  headset  is  easier  to  use. 

The  control  system  is  very  similar  to  many  video  games.  And,  considering  the 
age  group  of  those  who  will  be  using  it  if  the  military  adopts  it,  this  control 
scheme  will  already  feel  familiar.  The  only  change  I  would  recommend  is  an 


option  to  invert  the  Y-axis  of  the  camera  controls.  1 

I  would  prefer  the  ability  to  have  the  camera  axis  not  be  inverted  (up=up).  At 
times  the  camera  did  not  respond  to  my  commands  or  was  jumpy.  1 

Used  to  having  a  big  picture  with  a  little  map.  Also,  focusing  on  smallness  of 
picture  made  it  more  difficult  to  hit  waypoint  and  steer  camera.  1 

I  found  that  maintaining  situational  awareness  while  moving  to  be  difficult. 

The  controls  with  video  did  not  feel  smooth.  1 

Difficult  to  look  for  objects  outside  because  it  was  unclear  where  the  limits  of 
the  course  were.  A  zoom  function  would  have  helped  outdoors.  1 

It  would  be  easier  to  operate  if  when  you  pushed  up  on  the  controller  the 
camera  went  up  and  down  when  you  pressed  down.  1 

Moving  the  robot  was  the  most  difficult  part.  The  camera  was  moving  much 


too  quickly  (for  small  motions  as  well).  It  would  help  a  lot  if  I  could  make  the 
robot  have  slight  turns  of  its  head  at  first.  Then  its  head  moves  faster  as  I  push 


the  controller  harder  or  hold  it  down  longer.  Short,  jerky  movements  of  the 
head  took  a  lot  of  getting  used  to.  1 

The  controller’s  vision  is  very  sensitive.  I  think  the  sensitivity  could  become 
more  acute,  sharper.  1 

Controllers  were  too  sensitive  on  the  camera  side  of  things.  2 

Lagging  of  the  camera  was  hardest  part  of  navigation.  Sometimes  it  would 
freeze  up  and  it  can  lead  to  losing  orientation.  1 

For  both  iterations  the  video  freezing  made  the  recon  frustrating.  1 

Adding  360  degrees  ability  would  enable  the  robot  to  navigate  around  turns 
better.  Hard  to  judge  how  far  you  can  swing  out.  1 


Also,  there  was  a  huge  lag  between  the  image  on  my  screen  and  the  commands 
I  was  inputting  into  the  controls.  When  I  tell  the  robot  to  turn  its  head,  I  want 
to  see  the  picture  moving,  not  sudden  stops  and  starts  so  have  to  guess  how 
long  to  hold  the  button  down.  Function  to  “lock”  camera  to  a  target  no  matter 
where  I  maneuver.  1 

For  navigation,  it  was  very  difficult  to  tell  which  way  the  robot  was  travelling. 

The  only  way  I  could  follow  the  waypoint  map  was  to  watch  the  green  line  and 
red  “history”  line  on  the  overhead  map  while  travelling  down  the  route.  I  want 
to  scan  my  “eyes”  for  items  of  interest.  Its’  easy  to  get  separated  which 
direction  you’re  looking  around  compared  to  which  direction  you  are  driving. 

Those  two  horizontal  lines  that  represent  the  width  of  the  robot  just  don’t  cut  it 
because  as  you  turn  your  robot  head  farther,  they  are  out  of  sight.  It  would  1 
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Comments 


No.  of 
Responses 


help  a  lot  if  there  was  some  kind  of  gradient  for  the  entire  field  of  vision.  This 
way,  as  I  swivel  my  head,  it’s  easy  to  keep  driving  because  I  know  where  my 
body  is  I  relation  to  my  head. 

Also,  the  camera  -  there  needs  to  be  a  sound  effect  or  some  kind  of  indication 
or  flashing  light  or  something  to  confirm  I  took  a  picture.  Sometimes  a  small 
image  of  the  photo  I  took  would  appear,  and  when  there  were  several  pictures 
to  take  at  once  it  was  unclear  if  I  got  the  picture  or  if  the  button  didn’t  push  or 
what.  1 

The  picture  (video  feed)  from  robot  was  a  tiny  corner  of  the  screen  and  satellite 
map  was  most  of  the  screen.  I  was  straining  and  sitting  close  to  see  the  video, 
but  the  map  was  so  big.  Why  not  swap  them  so  have  a  large  video  to  see  the 
IEDs  more  clearly  and  I  can  navigate  with  a  smaller  route  map.  Or  the  route 
map  may  disappear  completely  and  only  appear  when  I  call  it  up  with  a  button 
on  the  controller.  Think  of  it  like  a  video  game:  I  should  get  options  to 
customize  how  I  want  the  display  to  look,  how  the  buttons  work  (steering, 
zoom  in/out).  The  more  options  I  have,  the  better  I  can  customize  the  robot’s 
performance  to  what  fits  me.  1 

Display  in  the  Laptop  was  too  small.  1 

Hard  to  keep  track  of  the  meanings  of  each  button,  and  the  display  on  the 
computer  is  confusing.  1 


TELEPRESENCE:  BUILDING 

N 

Mean 

Std.  Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Move  the  robot  in  the  correct 
direction 

16 

5.1875 

1.32759 

-1.171 

.564 

.922 

1.091 

Avoid  obstacles 

15 

4.8667 

1.18723 

-.299 

.580 

-.304 

1.121 

Avoid  potholes 

4 

4.5000 

1.00000 

-2.000 

1.014 

4.000 

2.619 

Assess  down  slopes  for 
navigability 

2 

4.0000 

1.41421 

Assess  side  slopes  for 
navigability 

2 

4.0000 

1.41421 

Identify  any  other  terrain 
features  that  makes  it  difficult 
to  maneuver  through  the 
terrain 

10 

5.4000 

.84327 

.389 

.687 

.370 

1.334 

Anticipate  whether  the  ground 
clearance  of  the  vehicle  will 
allow  negotiation  of  rugged 
terrain 

6 

5.5000 

1.04881 

.000 

.845 

-.248 

1.741 

Anticipate  whether  the  turn 
radius  of  the  vehicle  will 
allow  a  turn 

16 

4.0000 

1.63299 

.000 

.564 

-.458 

1.091 

Identify  if  the  robot  is  on  the 
correct  path 

12 

5.4167 

.90034 

-.152 

.637 

-.427 

1.232 
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TELEPRESENCE:  BUILDING 

N 

Mean 

Std.  Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Navigate  far  enough  ahead  to 
plan  route  in  advance 

15 

5.4667 

1.12546 

-.425 

.580 

.261 

1.121 

Navigate  well  enough  to  drive 
at  slowest  speeds 

15 

6.0000 

.75593 

.000 

.580 

-1.077 

1.121 

Navigate  well  enough  to  drive 
at  medium  speeds 

15 

5.9333 

1.09978 

-1.339 

.580 

2.449 

1.121 

Navigate  well  enough  to  drive 
at  fastest  speeds 

14 

5.5000 

1.28602 

-.380 

.597 

-.715 

1.154 

Finish  the  course  quickly 

16 

5.4375 

1.15289 

-.423 

.564 

-.189 

1.091 

Find  IEDs  and  other  objects 
of  interest 

16 

5.9375 

1.06262 

-1.386 

.564 

2.789 

1.091 

Navigate  to  the  next  waypoint 

10 

5.6000 

1.07497 

-.322 

.687 

-.882 

1.334 

Ability  to  map  the  bldg  (after 
recon) 

15 

5.8000 

1.08233 

-.328 

.580 

-1.126 

1.121 

Ability  to  take  pictures 

16 

5.9375 

1.28938 

-1.149 

.564 

.341 

1.091 

Being  aware  of  the  direction 
the  camera  is  pointed  when  it 
is  not  centered  on  the  front  of 
the  robot 

4 

4.7500 

1.25831 

-1.129 

1.014 

2.227 

2.619 

Make  robot  understand  you 

10 

5.8000 

1.03280 

-.272 

.687 

-.896 

1.334 

Ability  to  control  the  camera 
(viewing  direction) 

16 

5.5000 

1.59164 

-1.077 

.564 

.234 

1.091 

Maintain  situation  awareness 

16 

5.2500 

1.43759 

-.654 

.564 

-.121 

1.091 

Overall  ability  to  perform  this 
reconnaissance 

16 

5.5625 

1.03078 

-.191 

.564 

-.945 

1.091 

Move  the  robot  and  camera 
angle  at  the  same  time 

12 

4.8333 

1.85047 

-.949 

.637 

.415 

1.232 

Maintain  direction  to  way- 
point  while  searching  for 
items 

9 

4.7778 

1.85592 

-1.094 

.717 

.963 

1.400 

Laptop:  BUILDING 

N 

Mean 

Std.  Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Move  the  robot  in  the  correct 
direction 

16 

5.6250 

1.08781 

-.899 

.564 

.982 

1.091 

Avoid  obstacles 

16 

5.2500 

1.23828 

-1.023 

.564 

2.089 

1.091 

Avoid  potholes 

5 

5.4000 

1.14018 

.405 

.913 

-.178 

2.000 

Assess  down  slopes  for 
navigability 

3 

4.0000 

1.00000 

.000 

1.225 

Assess  side  slopes  for 
navigability 

3 

4.0000 

1.00000 

.000 

1.225 
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Laptop:  BUILDING 

N 

Mean 

Std.  Deviation 

Skewness 

Kurt  os  is 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Identify  any  other  terrain  features 
that  makes  it  difficult  to 
maneuver  through  the  terrain 

12 

5.8333 

1.19342 

-.392 

.637 

-1.446 

1.232 

Anticipate  whether  the  ground 
clearance  of  the  vehicle  will 
allow  negotiation  of  rugged 
terrain 

8 

4.6250 

1.30247 

-1.140 

.752 

1.652 

1.481 

Anticipate  whether  the  turn  radius 
of  the  vehicle  will  allow  a  turn 

16 

4.4375 

1.36473 

-.210 

.564 

-1.337 

1.091 

Identify  if  the  robot  is  on  the 
correct  path 

14 

5.2143 

1.31140 

-.458 

.597 

-.751 

1.154 

Navigate  far  enough  ahead  to 
plan  route  in  advance 

15 

5.3333 

1.11270 

.306 

.580 

-1.157 

1.121 

Navigate  well  enough  to  drive  at 
slowest  speeds 

15 

6.0000 

.92582 

-.623 

.580 

-.179 

1.121 

Navigate  well  enough  to  drive  at 
medium  speeds 

15 

5.8000 

1.01419 

-.493 

.580 

-.598 

1.121 

Navigate  well  enough  to  drive  at 
fastest  speeds 

14 

5.3571 

1.00821 

-.858 

.597 

1.211 

1.154 

Finish  the  course  quickly 

16 

5.2500 

1.23828 

-1.505 

.564 

2.597 

1.091 

Find  IEDs  and  other  objects  of 
interest 

16 

6.3125 

.70415 

-.537 

.564 

-.643 

1.091 

Navigate  to  the  next  waypoint 

8 

6.1250 

.83452 

-.277 

.752 

-1.392 

1.481 

Ability  to  map  the  bldg  (after 
recon) 

16 

5.8750 

1.20416 

-1.039 

.564 

.653 

1.091 

Ability  to  take  pictures 

16 

6.3125 

.79320 

-1.578 

.564 

3.902 

1.091 

Being  aware  of  the  direction  the 
camera  is  pointed  when  it  is  not 
centered  on  the  front  of  the  robot 

4 

4.7500 

1.89297 

-1.659 

1.014 

2.615 

2.619 

Make  robot  understand  you 

10 

6.1000 

.99443 

-1.085 

.687 

.914 

1.334 

Ability  to  control  the  camera 
(viewing  direction) 

16 

5.4375 

1.50416 

-.602 

.564 

-.974 

1.091 

Maintain  situation  awareness 

16 

5.1250 

1.14746 

-.274 

.564 

.249 

1.091 

Overall  ability  to  perform  this 
reconnaissance 

14 

5.6429 

.49725 

-.670 

.597 

-1.838 

1.154 

Move  the  robot  and  camera  angle 
at  the  same  time 

11 

4.9091 

1.86840 

-.965 

.661 

.385 

1.279 

Maintain  direction  to  way-point 
while  searching  for  items 

10 

5.0000 

1.76383 

-1.215 

.687 

2.245 

1.334 

No.  of 

Comments  Responses 

TELEPRESENCE:  BUILDING 

Great  for  field.  1 

This  was  the  easiest  and  less  frustrating  mission  out  of  all  of  them.  1 

Easier  than  outside.  1 

Inside  was  easier  to  navigate  and  find  IEDs  because  of  the  closeness.  1 
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No.  of 

Comments  Responses 

Vision  better  indoors;  robot  maneuvered  better  too.  1 

Much  easier  to  move  and  understand  information,  both  inside  and  out.  1 

Easier  to  find  objects  because  they  didn’t  blend  in  with  the  surroundings.  1 

Sometimes  the  visual  quality  was  bad;  hard  to  identify  items.  I  had  trouble 
seeing  the  guide  posts  on  occasion.  1 

Good  resolution  of  photographs/camera;  got  thru  course  well  enough.  1 

Responsive  controls  made  for  smoother  negotiation  of  course.  1 

The  controls  were  not  difficult  to  use  and  felt  very  straightforward.  1 

My  poor  skills  with  remote  controls  likely  contributed  the  most  to  slow  times  or 
missed  items.  1 

The  controller  occasionally  would  not  register  the  map  on  the  map  functions 
(zoom/item  scrolling).  My  one  complaint  is  the  head  piece  is  bulky  and, 
because  of  the  cords  in  the  rear,  sometimes  harder  to  turn  than  it  should  be.  1 

Given  smaller  size  of  buildings,  slightly  more  difficult  to  maneuver.  1 

Very  unsettling  because  of  feeling  room  was  smaller  than  it  was.  Recommend 
using  the  same  room  with  objects  being  moved.  Glitches  could  be  part  of  the 
nauseating  feeling.  1 

The  cameras  of  robot  were  much  more  functional  with  Telepresence;  much 
faster  and  easier  to  use,  but  very  difficult  to  distinguish  anything.  Why  not  just 
put  the  head  tilt  sensor  on  a  headband  (no  Telepresence  goggles)  and  let  me 
move  the  robot’s  eyes  by  moving  my  head  still;  but  I  would  much  rather  see 
all  the  video  come  over  the  Laptop  screen.  1 

Telepresence  goggles  have  other  imitations:  hard  to  see  when  looking  down  at 
my  feet  or  something  close  by  me.  And  the  power  cords  of  Telepresence 
goggles  are  heavy  and  make  it  hard  to  move  my  head  around.  The  cord  should 
be  hanging  from  the  ceiling  or  something.  1 

Taking  pictures  was  even  harder  because  I  kept  having  to  switch  screens  and 
select  the  category  from  a  list.  I  DON’T  WANT  TO  BE  BLIND  while  I’m 
selecting  from  a  list  or  watching  a  map.  Make  the  map  or  list  come  to  my 
video  screen  when  I  push  the  button,  so  I  can  still  operate  and  function  while  I 
am  navigating  and  taking  pictures.  1 

Having  a  large  back  end  makes  turning  through  doorways  difficult  and  you 
need  to  look  down  to  see  where  antennae  are  located.  1 

Bumpy  terrain  of  road  contributed  to  slight  motion  sickness,  but  easily  resolved 
by  “mono.”  1 

Overall,  no  need  for  “stereo”  camera  function  because  mono  provides  enough 
clarity  and  resolution  to  do  the  job  well.  1 

Camera  quality  still  made  it  hard  to  determine  what  an  object  was.  1 

Not  for  urban  operations.  1 

I  do  not  like  controlling  the  camera  with  the  headset.  I  have  always  used  a 
controller  to  maneuver  in  video  games  and  with  other  robots.  1 

Laptop;  BUILDING 

Camera  was  crisp  and  controlling  vehicle  was  quite  easy.  1 
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No.  of 

Comments  Responses 

Overall  this  system  was  easier  to  use  and  felt  more  natural  than  the  headset. 

The  one  other  recommendation  I  would  make  is  to  make  the  control  that 
moves  the  cameras  not  so  sensitive.  1 

Much  easier  to  see  and  feel  where  the  robot  is  and  how  to  maneuver  it.  1 

It  is  possible  to  move  the  robot  and  the  camera  at  the  same  time.  (I  wasn’t 
aware  that  t  was  possible,  so  I  did  not  use  that  feature.)  1 

The  “1”  button  function  to  re-center  the  “eyes”  to  the  body  was  very  useful.  I 
used  it  constantly  to  get  my  bearing  as  I  traveled.  Inside  it  let  me  look  around 
as  I  pleased  and  quickly  return  focus  to  the  front  without  having  to  wander  my 
eyes  back.  More  options  like  that  would  be  nice.  Like,  tapping  the  “eyes  left” 
stick  twice  would  make  the  robot  look  90  degrees  left  or  tapping  “move  right” 
twice  executes  “right  face”  turn  of  robot’s  body.  Do  a  brainstorm  of  all  the 
operations  that  would  be  beneficial  for  the  functionality  of  this  unit  and  think 


how  best  to  implement  them  in  the  form  of  user  friendly  controls.  1 

It  was  not  too  easy  to  make  this  robot  go  through  doorways  (and  look  for 
objects  at  same  time).  I  was  always  worried  about  banging  it  up  against  a  wall 
or  getting  stuck.  And  took  extra  time.  2 

Camera  freezing  was  the  only  problem.  1 

The  stereo  optic  view  presents  a  much  better  indoor  navigation,  and  makes  it 
easier  to  scan  for  items  compared  to  the  Laptop  system.  That  said,  in  my 
opinion  the  Laptop  is  superior  for  outdoors  navigation  because  it  retains  a  map 
to  orient  off  of.  1 

Found  it  easier  to  not  move  the  robot  and  camera  at  the  same  time.  It  was  too 
disorienting  to  drive  and  adjust  the  camera.  1 

Darkness  of  rooms  led  to  misidentification.  Otherwise,  would  like  a  bigger 
picture.  1 

The  controller  and  display  did  not  feel  like  one  unit.  1 


2.  Please  check  any  of  the  following  conditions  that  you  may  have  experienced  during  this  trial. 


NO.  OF  RESPONSES 

Telepresence 

Laptop 

a.  Eyestrain 

9 

3 

b.  Tunnel  vision 

2 

0 

c.  Headaches 

0 

0 

d.  Motion  sickness 

3 

1 

e.  Nausea 

1 

0 

f.  Disorientation 

4 

2 

g.  Dizziness 

0 

0 

h.  Competition  between  eyes  for  vision  of  different  scenes 
at  which  they  are  looking 

4 

0 

i.  Any  other  problems? 

2 

0 
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No.  of 

Comments  Responses 

TELEPRESENCE 

Dry  eyes.  1 

Eyestrain  caused  by  poor  image  quality.  1 

Eyestrain  due  to  objects  looking  even  smaller.  1 

Tunnel  vision:  Unable  to  see  some  areas  because  of  the  device.  1 

Motion  sickness  was  very  slight,  in  “stereo  vision.”  1 

h.  Cameras  not  aligned.  1 

Image  quality  is  a  NO  GO!  1 

A  little  blurry  for  first  5  seconds.  1 

Laptop 

No  comments. 

3.  What  are  your  comments  on  this  iteration? 

No.  of 

Comments  Responses 

TELEPRESENCE 

The  interface  I  think  will  be  very  useful  but  it  takes  a  little  time  to  get  used  to. 

However,  it  is  a  more  natural  way  to  view  things  and  moving  your  head  allows 
you  to  point  the  camera  exactly  where  you  need  to.  1 

A  zoom  function  would  be  helpful  with  identifying  objects.  1 

Made  it  sometimes  harder  to  look  down  and  see  the  antennas,  which  made  it 
harder  to  navigate  turns.  1 

I  was  unable  to  complete  this  iteration  while  using  the  stereo  vision.  1 

Laptop 

More  user-friendly.  Worked  better  in  this  mode  and  it  was  also  much  easier  to 
stay  oriented  to  where  you  are.  Whereas,  with  the  headset  you  could  become 
disoriented  much  more  easily.  While  the  headset  seems  cooler,  this  system  is 
the  easier  to  use  which  is  what  matters  in  a  combat  environment.  1 

Camera  system  works  really  well  overall.  It  seemed  to  be  more  inside  the 
building  than  outside.  The  coupling  of  the  map  and  red  and  green  lines  make 
it  easier  to  orient  yourself  to  the  camera.  The  re-center  button  is  a  huge  help 
for  quick  correction.  1 

After  rebooting  the  system,  I  photographed  and  labeled  three  soda  can  IEDs  as 
cell  phones/PDAs.  1 
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A-4  -  Experiment  1:  End  of  Experiment  Questionnaire  Results 


1.  Using  the  scale  below,  please  rate  the  training  you  received  in  the  following  areas. 

1  2  3  4  5  6  7 

Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely  good 


N 

Mean 

Std.  Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

TELEPRESENCE 

Completeness  of  introductory 
training 

16 

6.31 

.873 

-1.397 

.564 

2.016 

1.091 

Comprehension  of  overall 
concept  of  the  robot 

16 

6.31 

1.014 

-2.478 

.564 

7.791 

1.091 

Outdoors:  How  to  drive 

16 

6.19 

1.047 

-1.219 

.564 

.546 

1.091 

Outdoors:  Time  provided  to 
practice 

16 

6.13 

1.025 

-1.129 

.564 

.492 

1.091 

Outdoors:  How  to  perform 
reconnaissance 

16 

5.81 

1.167 

-1.025 

.564 

.833 

1.091 

Building:  How  to  drive 

16 

6.19 

.834 

-.391 

.564 

-1.443 

1.091 

Building:  Time  provided  to 
practice 

16 

6.06 

1.181 

-1.521 

.564 

2.057 

1.091 

Building:  How  to  perform 
reconnaissance 

16 

6.13 

.957 

-.798 

.564 

-.235 

1.091 

Overall  training  evaluation 

16 

6.13 

.957 

-.798 

.564 

-.235 

1.091 

Laptop 

Completeness  of  introductory 
training 

16 

6.31 

.873 

-1.397 

.564 

2.016 

1.091 

Comprehension  of  overall 
concept  of  the  robot 

16 

6.44 

.629 

-.653 

.564 

-.321 

1.091 

Outdoors:  How  to  drive 

16 

6.25 

.775 

-.492 

.564 

-1.062 

1.091 

Outdoors:  Time  provided  to 
practice 

16 

6.06 

1.181 

-1.521 

.564 

2.057 

1.091 

Outdoors:  How  to  perform 
reconnaissance 

16 

6.13 

.957 

-.798 

.564 

-.235 

1.091 

Building:  How  to  drive 

16 

6.19 

.834 

-.391 

.564 

-1.443 

1.091 

Building:  Time  provided  to 
practice 

16 

6.00 

1.317 

-1.603 

.564 

1.929 

1.091 

Building:  How  to  perform 
reconnaissance 

16 

6.25 

.931 

-1.133 

.564 

.677 

1.091 

Overall  training  evaluation 

16 

6.38 

.806 

-.845 

.564 

-.838 

1.091 

No.  of 

Comments  Responses 

Excellent.  1 

Very  comprehensive.  2 

Training  was  very  specific  and  well-explained.  1 
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Overall,  very  good.  2 

Ease  of  use  was  good.  1 

Training  was  very  logical  and  easy  to  comprehend.  1 

Instructions  were  clear.  1 

The  robot  is  very  easy  to  operate.  1 

Controls  were  easy  to  use.  1 

Conditions  and  standards  were  well  explained  and  understood.  1 

I  felt  I  had  received  the  proper  training  to  perform  all  the  duties.  1 

I  really  like  the  second  trial  with  the  Telepresence;  much  easier.  My  eyes  were 
less  strained  to  see  what  I  was  looking  at.  1 

The  Laptop  offers  a  format  most  people  are  used  to  using,  so  it  is  easier  to  take 
the  new  technology.  Whereas,  the  HMD  is  foreign  and  not  as  familiar  to  new 
user.  1 

I  prefer  the  Laptop  by  far.  1 

I  found  it  more  difficult  to  control  the  camera  using  Telepresence  than  using  the 
Laptop.  1 

A  little  more  instruction  on  how  to  turn  and  maneuver  the  robot  and  use  of  the 
lines  would  have  been  helpful.  1 

Very  much  the  same.  1 

Some  variables  make  it  more  inclined  for  others  to  get  motion  sickness  while 
using  the  Telepresence.  1 

Laptop  screen  was  hard  to  use  due  to  the  small  picture  and  camera  controller.  1 

Would  like  a  larger  viewing  screen  and  a  smaller  GPS  map.  Would  make 
targets  easier  to  identify.  1 

Zoom  function  would  have  been  nice.  1 


2.  What  were  the  easiest  and  hardest  training  tasks  to  learn? 

Comments 

Easiest 

To  swivel  the  camera  to  view  items  of  interest. 

How  to  use  the  camera  to  conduct  recon. 

Taking  a  photo. 

How  to  move  the  robot  and  look  for  items  listed. 

Moving  the  robot  with  the  Laptop. 

Driving. 

Driving  with  the  Laptop. 

Driving  indoors. 

Using  the  controls. 

Linding  waypoints  and  your  sense  of  direction  was  easiest. 

Hardest 

Moving  your  head  in  sync  with  the  robot,  or  just  looking  around  in  general 
while  driving  the  robot  in  a  specific  direction. 


No.  of 


Responses 


1 

1 

1 

2 

1 

1 

1 

1 

2 

1 


1 
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No.  of 

Comments  Responses 

To  learn  to  maneuver  the  robot,  especially  around  or  near  obstacles  to 
doorways.  1 

Driving  the  robot.  2 

Driving  with  Laptop.  1 

Trying  to  move  the  camera.  1 

Moving  the  camera  with  the  Laptop  system.  1 

Centering  the  camera  with  the  Telepresence.  1 

Using  the  headset  and  getting  oriented  with  it.  1 

Using  the  virtual  headset  to  move  camera.  1 

Controlling  the  camera  and  the  robot  at  the  same  time  using  Telepresence.  1 

Multifunction  tasks:  selecting  which  object  from  list  before  taking  picture, 
simultaneous  camera  move/robot  move.  1 

Identifying  the  objects.  1 

Controls  took  awhile  to  get  used  to.  1 

Remembering  the  button  functions  and  to  make  sure  to  call  the  waypoints.  1 

Getting  used  to  the  HMD.  1 

Backward  up  and  down  was  difficult  with  the  controller.  1 

3.  What  are  your  overall  comments  on  the  training? 

Excellent  training.  2 

Good.  3 

It  was  interesting  and  fun  to  participate  in  the  research  study.  1 

Very  simple;  perfectly  adequate.  1 

Easy  to  understand  and  operate.  1 

Visual  guide  for  the  controller  was  excellent.  1 

Adequate  to  orient  the  operator  to  how  to  use  the  controls  and  what  tasks 
needed  to  be  accomplished.  1 

Will  help  the  military  to  save  lives.  1 

It  was  adequate,  but  helpful  to  learn  and  familiarize  myself  with  the  controls.  1 

Laptop  unit  was  simple  and  easy  to  quickly  identify  targets.  1 
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4.  Using  the  scale  below,  please  rate  the  following  characteristics  of  each  control  system. 


1  2  3  4  5  6  7 

Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely  good 


CHARACTERISTICS 

N 

Mean 

Std.  Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

TELEPRESENCE 

Resolution  (Clarity) 

16 

5.25 

1.390 

-.850 

.564 

.798 

1.091 

Size  of  objects  appearing  in  the 
display 

16 

5.25 

1.483 

-.630 

.564 

.008 

1.091 

Ability  to  adjust  display 

15 

4.60 

1.502 

-.659 

.580 

1.060 

1.121 

Comfort  of  viewing 

16 

4.75 

1.183 

-.276 

.564 

.971 

1.091 

Display  brightness 

16 

5.25 

1.342 

-1.088 

.564 

1.221 

1.091 

Display  glare 

16 

5.63 

1.310 

-.210 

.564 

-1.793 

1.091 

Contrast  between  objects  on  the 
driving  display 

15 

5.53 

1.302 

-.532 

.580 

-.837 

1.121 

Display  color 

15 

5.47 

.990 

-.149 

.580 

-.844 

1.121 

Comfort  of  using  display 

16 

5.31 

1.352 

-.838 

.564 

.900 

1.091 

Number  of  controls 

16 

5.88 

1.025 

-.571 

.564 

-.592 

1.091 

Control  locations 

16 

6.25 

.856 

-1.274 

.564 

1.907 

1.091 

Size  of  individual  controls 

16 

6.25 

.683 

-.358 

.564 

-.592 

1.091 

Complexity  of  controls 

16 

6.25 

.775 

-.492 

.564 

-1.062 

1.091 

Ability  to  use  controls  without 
activating  other  controls 

16 

5.94 

1.181 

-1.251 

.564 

1.289 

1.091 

Size  of  entire  control  unit 

16 

6.56 

.629 

-1.183 

.564 

.633 

1.091 

Adequacy  of  this  control  unit 
for  teleoperation 

16 

6.00 

.966 

-2.028 

.564 

6.127 

1.091 

Ease  of  controlling  camera 
(viewing  direction) 

16 

5.38 

1.784 

-.893 

.564 

.551 

1.091 

Overall  assessment  of  this 
control  unit 

16 

5.75 

1.065 

-.189 

.564 

-1.183 

1.091 

Laptop 

Resolution  (Clarity) 

16 

5.81 

.544 

-.189 

.564 

.555 

1.091 

Size  of  objects  appearing  in  the 
display 

16 

5.38 

.806 

.027 

.564 

-.130 

1.091 

Ability  to  adjust  display 

15 

4.87 

1.187 

-.004 

.580 

-.791 

1.121 

Comfort  of  viewing 

16 

5.81 

.911 

-.797 

.564 

.412 

1.091 

Display  brightness 

16 

5.81 

1.047 

-.375 

.564 

-.948 

1.091 

Display  glare 

16 

5.13 

1.455 

.051 

.564 

-1.485 

1.091 

Contrast  between  objects  on  the 
driving  display 

15 

5.47 

.640 

-.802 

.580 

-.127 

1.121 

Display  color 

15 

5.67 

.724 

-.676 

.580 

.948 

1.121 

Comfort  of  using  display 

16 

5.69 

1.195 

-1.445 

.564 

1.841 

1.091 

Number  of  controls 

16 

5.50 

1.155 

-.297 

.564 

-1.411 

1.091 

Control  locations 

16 

5.94 

1.181 

-1.251 

.564 

1.289 

1.091 

Size  of  individual  controls 

16 

6.13 

.719 

-.192 

.564 

-.821 

1.091 

Complexity  of  controls 

16 

5.88 

1.147 

-.936 

.564 

.939 

1.091 
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CHARACTERISTICS 

N 

Mean 

Std.  Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

Ability  to  use  controls  without 
activating  other  controls 

16 

6.12 

1.025 

-1.129 

.564 

.492 

1.091 

Size  of  entire  control  unit 

16 

6.56 

.629 

-1.183 

.564 

.633 

1.091 

Adequacy  of  this  control  unit 
for  teleoperation 

15 

6.20 

.561 

.112 

.580 

.378 

1.121 

Ease  of  controlling  camera 
(viewing  direction) 

16 

5.31 

1.352 

-.838 

.564 

-.529 

1.091 

Overall  assessment  of  this 
control  unit 

16 

5.81 

.750 

.334 

.564 

-1.004 

1.091 

No.  of 


Comments  Responses 

TELEPRESENCE 

Telepresence  is  better  for  controlling  the  robot,  but  mainly  only  for  indoor 
situations.  1 

Felt  more  comfortable  and  I  was  less  pressured  for  time  because  of  that.  1 

Better  for  indoors.  1 

Better  for  observation,  but  sometimes  disorienting.  1 

Felt  like  I  was  in  the  robot’s  head,  looking  for  IEDs.  1 

Not  very  much  better  until  the  image  quality  from  the  Telepresence  glasses  is 
addressed.  1 

Cords  of  Telepresence  are  heavy.  1 

I  would  never  want  to  use  in  the  field  if  I  had  access  to  a  control  and  display 
system  like  the  Laptop.  1 

Laptop 

Easier  to  use  and  felt  more  natural.  1 

More  user-friendly.  1 

Good  for  outdoors.  1 

Laptop  did  not  make  me  feel  nauseated.  1 

Too  small.  1 

Resolution  problems  and  clarity  issues  (none  of  which  were  present  with 
Telepresence).  1 

General  Comments: 

Great.  May  be  its  too  simple.  1 

Make  video  screen  larger.  1 

Needs  more  adjustment  options.  1 

Both  comfortable  for  viewing.  1 

Needs  to  be  mounted  from  ceiling.  1 

“Change  map  button”  is  close  to  “vibrate  button.”  1 

Depends  on  the  situation  as  to  which  system  is  better.  1 

5.  Using  the  scale  below,  please  rate  each  control  system  for  the  following  tasks. 


Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely  good 
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TASKS 

N 

Mean 

Std.  Deviation 

Skewness 

Kurtosis 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Statistic 

Std.  Error 

TELEPRESENCE 

Avoiding  obstacles 

16 

5.31 

.946 

-.187 

.564 

1.950 

1.091 

Assessing  terrain  for 
navigability 

16 

5.81 

1.109 

-.588 

.564 

-.860 

1.091 

Driving  straight  route 

16 

5.25 

1.238 

-.060 

.564 

-.962 

1.091 

Driving  multiple  waypoints 

16 

5.44 

1.263 

-.293 

.564 

-.872 

1.091 

Looking  for  objects  of  interest 

16 

5.50 

1.461 

-.734 

.564 

-.578 

1.091 

Identifying  objects 

16 

5.44 

1.632 

-1.557 

.564 

2.642 

1.091 

Scanning  surroundings 

16 

5.75 

1.342 

-.615 

.564 

-.809 

1.091 

Maneuvering  comers 

16 

5.50 

1.265 

-.226 

.564 

-.790 

1.091 

Maintaining  situation  awareness 

16 

5.31 

1.621 

-1.331 

.564 

2.213 

1.091 

Overall  Building  reconnaissance 

16 

5.81 

1.424 

-1.365 

.564 

2.044 

1.091 

Overall  Outdoor  reconnaissance 

16 

5.06 

1.389 

-.126 

.564 

-.993 

1.091 

Laptop 

Avoiding  obstacles 

16 

6.13 

.719 

-.192 

.564 

-.821 

1.091 

Assessing  terrain  for 
navigability 

16 

6.06 

.772 

-.113 

.564 

-1.194 

1.091 

Driving  straight  route 

16 

6.13 

.957 

-.798 

.564 

-.235 

1.091 

Driving  multiple  waypoints 

16 

6.06 

.680 

-.074 

.564 

-.489 

1.091 

Looking  for  objects  of  interest 

16 

5.69 

.946 

-1.430 

.564 

3.626 

1.091 

Identifying  objects 

16 

5.75 

.577 

.000 

.564 

-.066 

1.091 

Scanning  surroundings 

16 

5.56 

1.094 

-.356 

.564 

.549 

1.091 

Maneuvering  comers 

16 

5.88 

.806 

.245 

.564 

-1.368 

1.091 

Maintaining  situation  awareness 

16 

5.87 

1.088 

-1.143 

.564 

1.889 

1.091 

Overall  Building  reconnaissance 

16 

5.63 

1.025 

-.810 

.564 

1.645 

1.091 

Overall  Outdoor  reconnaissance 

16 

6.00 

.816 

.000 

.564 

-1.467 

1.091 

No.  of 

Comments  Responses 

TELEPRESENCE 

Great  for  outside.  1 

Much  easier  to  tell  what  the  objects  were  and  where  they  were.  1 

Not  bad.  1 

Telepresence  seems  better,  but  the  system  needs  some  refinement  before  it  is 
the  clear  winner.  1 

If  it  had  a  map  integrated,  would  be  perfect.  1 

Waypoint  identification  difficult  due  to  the  need  to  go  back  and  forth  between 
two  maps.  1 

Harder  to  use  indoors.  1 

Laptop 

Superior  outdoors,  primarily  because  you  can  use  the  map  to  orient  the  robot’s 
direction.  1 
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No.  of 

Comments  Responses 

TELEPRESENCE 

Easier  to  use  and  control.  1 

More  familiar  and  a  better  interface  for  the  given  task.  1 

Would  have  been  nice  and  easier  to  use  the  Laptop  if  the  map  was  the  smaller, 
inset  image  and  the  camera  image  was  the  main  image.  1 

Laptop  was  too  small  and  unclear  for  fast  and  accurate  identification.  1 

General  comments: 

Stereoscopic  vision  was  not  useful  to  me  at  all  on  this  system.  1 

Very  hard  to  identify  objects  on  inner  rooms.  1 


6.  Which  control  system  do  you  prefer  (why,  and  under  what  conditions)? 


Number  of  Responses 

Telepresence  (HMD) 

Laptop  (Laptop) 

9 

6 

*one  no  response 


No.  of 

Comments  Responses 

TELEPRESENCE 

Superior  for  indoor  navigation,  especially  with  the  depth  perception.  1 

Stereo  display  was  helpful.  1 

I  prefer  the  Telepresence  in  the  building  environment.  Control  of  vision  and 
navigation  were  superior  to  Laptop,  but  the  video  image  was  much  easier  to 
identify  items  of  interest  than  the  Telepresence  goggles.  Higher  quality  goggle 


image  is  definitely  needed.  1 

Had  a  much  easier  time  doing  surveillance  and  scanning.  1 

Easy  during  outdoor,  and  especially  the  indoor  was  easier  to  see  and  tell  what 
the  objects  were  and  where  they  were  on  the  screen.  1 

Easier  on  outside  because  you  can  move  while  scanning  for  explosives.  1 

A  more  natural  way  to  search,  leaving  the  mind  able  to  focus  on  the  search  and 
not  the  controls.  1 

Ability  to  control  the  view  so  easily.  1 

Not  good  on  inside.  1 

Had  to  be  re-calibrated  during  the  exercise.  1 

Laptop 

The  Laptop  was  more  user-friendly  and  seemed  to  have  a  somewhat  clearer 
display.  1 

Lamiliarity.  1 

Easier  to  use  because  of  the  access  to  the  menu  without  switching  back  and 
forth.  1 

Joystick  control  of  the  camera  offers  steadier  picture.  1 

Crisper  picture.  1 

Easier  not  to  become  disoriented  than  the  Telepresence.  1 
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No.  of 

Comments  Responses 

Better  for  outside.  1 

Display  picture  was  too  small  and  hard  to  see.  1 

Laptop  inside  because  the  objects  are  more  clear.  1 

7.  What  suggestions  do  you  have  for  ways  to  increase  the  effectiveness  of  the  following? 

No.  of 

Comments  Responses 

TELEPRESENCE 

Very  good.  1 

Work  out  glitches  with  software  to  hardware  to  prevent  video  freezing. 

Commands  not  being  received  and  fix  the  drift  of  what  is  “facing  straight 
ahead”  on  the  headset  with  the  cameras.  Forward  should  always  be  forward.  1 

Put  the  list  of  objects  to  identify  on  both  views  (the  map  and  the  real  time 
view).  1 

Both  cameras  should  be  adjusted  to  show  a  better  display  with  close  objects.  1 

Have  the  map  inset  into  screen  instead  of  having  to  switch  between  the  two.  1 

Integrate  a  map  in  the  upper  left  corner  or  right  corner,  toggled  by  the  map 
button,  to  aid  in  navigation.  1 

More  contrast  between  colors.  1 

An  in/out  zoom  for  the  cameras.  3 

A  wider  peripheral  vision  would  help.  1 

A  light  headpiece  would  influence  more  head  turning.  1 

Better  resolution  of  the  picture.  5 

Make  the  view  not  so  jumpy  when  the  head  is  moved.  1 

More  stabilization  of  headset  so  that  it  does  not  require  re-calibration.  1 

Controls  feel  touchy.  1 

Could  be  enhanced  by  not  making  it  as  sensitive  to  the  motion  so  it  is  easier  to 
keep  straight.  1 

“Stereo”  is  unnecessary  because  “mono”  is  more  than  adequate  for 
accomplishing  the  mission.  1 

May  need  a  small  camera  on  back  to  see  reverse.  1 

Attach  the  cord  so  that  it  does  not  slip  or  become  caught,  jerking  the 
controller’s  head.  1 

Laptop  (Laptop) 

Hard  to  improve  because  it  does  not  offer  what  the  Telepresence  does.  1 

Have  the  camera  higher  off  the  ground  to  see  farther/more  clearly.  1 

Using  keyboard  controls  rather  than  the  joystick  (or  combination  of  both).  1 

Make  the  controls  more  sensitive  to  slight  movements  on  the  controller.  1 

Reprogram  the  control  unit.  1 

Increase  the  size  of  the  camera  display  image.  5 

Make  the  video  large  and  the  waypoint  map  smaller.  2 

Make  pre-defined  functions  (combos)  that  can  be  entered  on  the  controller,  such 
as  “right  face,”  “left  face,”  etc.  1 
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Comments 


TELEPRESENCE 


No.  of 
Responses 


Very  good.  1 

Work  out  glitches  with  software  to  hardware  to  prevent  video  freezing. 

Commands  not  being  received  and  fix  the  drift  of  what  is  “facing  straight 
ahead”  on  the  headset  with  the  cameras.  Forward  should  always  be  forward.  1 

Put  the  list  of  objects  to  identify  on  both  views  (the  map  and  the  real  time 
view).  1 

Both  cameras  should  be  adjusted  to  show  a  better  display  with  close  objects.  1 

Have  the  map  inset  into  screen  instead  of  having  to  switch  between  the  two.  1 

Integrate  a  map  in  the  upper  left  corner  or  right  corner,  toggled  by  the  map 
button,  to  aid  in  navigation.  1 

More  contrast  between  colors.  1 

An  in/out  zoom  for  the  cameras.  3 

A  wider  peripheral  vision  would  help.  1 

A  light  headpiece  would  influence  more  head  turning.  1 

Better  resolution  of  the  picture.  5 

Make  the  view  not  so  jumpy  when  the  head  is  moved.  1 

More  stabilization  of  headset  so  that  it  does  not  require  re-calibration.  1 

Controls  feel  touchy.  1 

Could  be  enhanced  by  not  making  it  as  sensitive  to  the  motion  so  it  is  easier  to 
keep  straight.  1 

“Stereo”  is  unnecessary  because  “mono”  is  more  than  adequate  for 
accomplishing  the  mission.  1 

May  need  a  small  camera  on  back  to  see  reverse.  1 

Attach  the  cord  so  that  it  does  not  slip  or  become  caught,  jerking  the 
controller’s  head.  1 

Laptop  (Laptop) 

Hard  to  improve  because  it  does  not  offer  what  the  Telepresence  does.  1 

different  options  for  up  and  down. 

Make  the  controls  not  as  sensitive,  invert  the  y-axis. 

Could  be  approved  by  allowing  for  a  broader  range  of  camera  angles;  perhaps 
360°  view. 

Better  resolution;  clearer. 

Ability  to  zoom  the  camera. 
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A-5  -  Experiment  2:  Demographic  Results  (N  =  22) 


MOS 

RANK 

AGE 

DUTY  POSITION 

09S- 

17 

E-4-4 

26  years 

OCS  -  22 

15Q- 

1 

E-5  -  15 

(mean) 

92A  - 

1 

NR -3 

1 .  How  long  have  you  served  in  the  military?  12  months  (mean) 

2.  How  long  have  you  had  an  infantry-related  job?  5.5  months  (mean)  (N  =  2) 

3.  How  long  have  you  been  deployed  overseas?  15  months  (mean)  (N  =  3) 

4.  How  long  have  you  been  deployed  in  a  combat  area?  12  months  (mean)  (N  =  1) 

5.  With  which  hand  do  you  most  often  write?  21  Right  1  Left 

6.  With  which  hand  do  you  most  often  fire  a  weapon?  20  Right  _2_  Left 

7.  a.  Do  you  wear  prescription  lenses?  9  No  13  Yes 

b.  If  so,  which  do  you  most  often  wear?  8  Glasses  4  Contacts  1  NR 

c.  Which  is  your  dominant  eye?  18  Right  3_  Left  J_  NR 

d.  Do  you  have  any  vision-related  problems?  16  No  _3_Yes  3  NR 
If  so,  what?  20/40  in  right  eye,  near-sightedness  (2), 

8.  Do  you  have  any  hearing-related  problems?  17  No  J_Yes  4  NR 
If  so,  what?  Slight  hearing  loss  in  right  ear. 

9.  Have  you  ever  experienced  motions  sickness?  14  No  _5_  Yes  3  NR 
If  so,  under  what  conditions?  Riding  in  a  car,  roller  coaster  (2),  boats  (2) 
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10. a.  Have  you  ever  used  a  robotic  system?  20  No  0  Yes  2  NR 


b.  If  so,  what  type? 

c.  Please  describe  the  conditions  under  which  you  used  the  robotic  system. 

11.  Please  rate  your  skill  level  for  each  of  the  following  activities? 

None  Beginner  Intermediate  Expert 

12  3  4 


ACTIVITY 

MEAN  RESPONSE 

Operating  ground  unmanned  vehicles 

1.14 

Operating  aerial  vehicles 

1.18 

Target  detection  and  identification 

1.73 

Playing  commercial  video  games 

2.82 

Training  with  Army  video  simulations 

1.86 

12.  How  many  years  of  experience  do  you  have  operating  a  robot 

21  None  1  Less  than  1  year  0  1-2  years  0  2-5  years  0  More  than  5  years 

13.  How  many  hours  of  robot  simulator  experience  do  you  have?  14  None 

22  Less  than  2  hours  0  2-5  hours  0  5-10  hours  0  More  than  10  hours 


14.  How  many  times  have  you  been  deployed  and  used  an  unmanned  vehicle  for  EOD  missions? 
22  None  _0_  1-2  times  _0_  3-4  times  _0_  5-6  times  _0_  7  times  or  more 

15.  How  often  do  you  play  video  games? 

1  Never  _5_  5-11  times  per  year  J5_  1-3  times/month  9_  1-6  times/week  J_  Daily 

16.  Which  gaming  system  do  you  plan  most  often?  XBox  (6),  Sony  PSP,  Playstation  (8), 
iPhone,  PC/Laptop  (4),  Wii  (4),  Nintendo  DS  (3) 
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A-6  -  Experment  2:  Performance  Results 


Audio  Task:  Time  to  find  target 


N 

Minimum 

Maximum 

Mean 

Std.  Deviation 

Skewness 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

Target  B 

BT_secl 

16 

29.00 

196.00 

64.6250 

41.08183 

2.453 

.564 

BMJ_secl 

16 

55.00 

562.00 

141.6875 

126.78418 

2.759 

.564 

BMH_secl 

16 

33.00 

196.00 

85.8750 

43.19394 

.969 

.564 

Target  D 

DT_sec  1 

16 

28.00 

153.00 

75.1875 

36.11319 

1.208 

.564 

DMJ_secl 

16 

56.00 

272.00 

134.8125 

64.76132 

1.225 

.564 

DMH_sec  1 

16 

46.00 

248.00 

107.3750 

53.02185 

1.599 

.564 

Target  H 

HT_sec  1 

16 

28.00 

83.00 

46.8125 

16.16671 

.817 

.564 

HMJ_secl 

16 

37.00 

469.00 

95.6250 

104.10307 

3.467 

.564 

HMH_sec  1 

16 

23.00 

165.00 

68.3750 

34.80397 

1.302 

.564 

Target  J 

JT_secl 

16 

37.00 

197.00 

98.2500 

44.35689 

1.019 

.564 

JMJ_secl 

16 

64.00 

258.00 

139.2500 

63.49541 

.459 

.564 

JMH_secl 

16 

44.00 

231.00 

103.5000 

48.32805 

1.534 

.564 

Target  G 

GT_sec  1 

16 

24.00 

120.00 

47.5625 

23.11412 

2.180 

.564 

GMJ_secl 

16 

25.00 

355.00 

83.2500 

77.05626 

3.241 

.564 

GMH_sec  1 

16 

30.00 

259.00 

70.5625 

56.60385 

2.719 

.564 

Target  E 

ET_sec  1 

16 

26.00 

92.00 

57.3125 

19.72382 

.129 

.564 

EMJ_sec  1 

16 

39.00 

234.00 

89.1875 

55.94428 

1.730 

.564 

EMH_secl 

16 

29.00 

254.00 

93.3125 

57.44414 

1.642 

.564 

Valid  N  (listwise) 

16 
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Descriptive  Statistics 


Mean  Std.  Deviation  N 


97.3958  60.39502  16 

105.7917  41.39921  16 

70.2708  44.22170  16 


Jtime  113.6667  48.13376  16 

Gtime  67.1250  37.27871  16 

Etime  79.9375  38.16770  16 


Audio  Search  times  by  target 


Time  to  Approach  by  Display  and  Target 


Mean 

Std.  Deviation 

N 

BT_sec2 

89.6250 

39.34950 

16 

BMJ_sec2 

164.6250 

130.33591 

16 

BMH_sec2 

112.5625 

54.42545 

16 

DT_sec2 

112.1875 

93.39252 

16 

DMJ_sec2 

154.3750 

66.80906 

16 

DMH_sec2 

128.1250 

56.34640 

16 

HT_sec2 

69.0000 

23.44355 

16 

HMJ_sec2 

124.6250 

120.17591 

16 

HMH_sec2 

90.1250 

44.86703 

16 

JT_sec2 

122.7500 

44.32682 

16 

JMJ_sec2 

172.5625 

80.43463 

16 

JMH_sec2 

130.5625 

61.54886 

16 

GT_sec2 

65.9375 

27.24817 

16 

GMJ_sec2 

101.8750 

80.23538 

16 

GMH_sec2 

81.8125 

53.83954 

16 

ET_sec2 

75.8125 

22.31358 

16 

EMJ_sec2 

111.3125 

55.38438 

16 

EMH_sec2 

108.5000 

59.98555 

16 
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Mean  Percentage  correctly  Identified  by  display 
and  target 


Mean 

Std.  Deviation 

N 

BT_ID 

.9375 

.25000 

16 

BMJ_ID 

.2500 

.44721 

16 

BMH_ID 

.3125 

.47871 

16 

DT_ID 

.6250 

.50000 

16 

DMJ_ID 

.6250 

.50000 

16 

DMH_ID 

.5625 

.51235 

16 

HT_ID 

.9375 

.25000 

16 

HMJ_ID 

.8750 

.34157 

16 

.8125 

.40311 

16 

JT_ID 

.9375 

.25000 

16 

JMJ_ID 

.6875 

.47871 

16 

JMH_ID 

.6875 

.47871 

16 

GT_ID 

.8750 

.34157 

16 

GMJ_ID 

.6875 

.47871 

16 

GMH_ID 

.8125 

.40311 

16 

ET_ID 

.8750 

.34157 

16 

EMJ_ID 

.5000 

.51640 

16 

EMH_ID 

.7500 

.44721 

16 

Mean  Distance  to  Target  by  Display  and  Target 


JT_dist 

49.5000 

11.66803 

15 

JMJ_dist 

45.2000 

14.82102 

15 

JMH_dist 

46.7167 

13.80860 

15 

GT_dist 

22.3000 

4.09943 

15 

GMJ_dist 

20.7633 

4.58579 

15 

GMH_dist 

22.6833 

5.02624 

15 

ET_dist 

26.3667 

1.90363 

15 

EMJ_dist 

27.9667 

8.36333 

15 

EMH_dist 

27.5833 

7.52654 

15 

Mean  driving  errors  by  Display  and  Target 


Mean 

Std.  Deviation 

N 

BT_drive 

.0000 

.00000 

15 

BMJ_drive 

.8000 

1.47358 

15 

BMH_drive 

.2000 

.56061 

15 

DT_drive 

.0667 

.25820 

15 

DMJ_drive 

.6667 

1.17514 

15 

DMH_drive 

.4000 

.73679 

15 

HT_drive 

.2000 

.41404 

15 

HMJ_drive 

.5333 

.74322 

15 

HMH_drive 

.5333 

.74322 

15 

JT_drive 

.8000 

.86189 

15 

JMJ_drive 

.8000 

.94112 

15 

JMH_drive 

.4000 

.63246 

15 

GT_drive 

.3333 

.48795 

15 

GMJ_drive 

.6667 

1.29099 

15 

GMH_drive 

.2667 

.59362 

15 

ET_drive 

.2667 

.45774 

15 

EMJ_drive 

.2000 

.56061 

15 

EMH_drive 

.3333 

.48795 

15 

N 


Minimum  Maximum 


Mean 


Std.  Deviation 


Skewness 


Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Statistic 

Std.  Error 

T_TLX_mental 

17 

1.00 

15.00 

6.7647 

3.89759 

.646 

.550 

T_TLX_physical 

16 

1.00 

11.00 

3.4375 

2.78014 

1.608 

.564 

T_TLX_temporal 

16 

1.00 

19.00 

6.7500 

5.36035 

1.553 

.564 

T_TLX_performance 

16 

11.00 

20.00 

15.8750 

2.52653 

-.577 

.564 

T_TLX_effort 

16 

2.00 

17.00 

7.6250 

4.77319 

.728 

.564 

T_TLX_frustration 

16 

1.00 

13.00 

4.3125 

3.82481 

1.483 

.564 

MH_TLX_mental 

16 

2.00 

20.00 

9.9375 

4.94596 

.137 

.564 

MH_TLX_physical 

16 

1.00 

16.00 

4.4375 

4.04918 

1.881 

.564 

MH_TLX_temporal 

16 

1.00 

19.00 

7.5625 

5.54940 

1.025 

.564 

MH_TLX_performance 

16 

8.00 

19.00 

14.1250 

3.38378 

-.229 

.564 

MH_TLX_effort 

16 

3.00 

17.00 

8.9375 

4.50879 

.426 

.564 

MH_TLX_frustration 

16 

2.00 

15.00 

7.3125 

4.48284 

.658 

.564 

MJ_TLX_mental 

16 

2.00 

20.00 

12.6875 

4.71478 

-.788 

.564 

MJ_TLX_physical 

16 

1.00 

18.00 

5.8125 

4.81966 

1.311 

.564 

MJ_TLX_temporal 

16 

1.00 

19.00 

8.9375 

5.91573 

.292 

.564 

MJ_TLX_performance 

16 

3.00 

19.00 

11.2500 

4.75395 

.150 

.564 

MJ_TLX_effort 

16 

3.00 

18.00 

11.6875 

4.23822 

-.743 

.564 

MJ_TLX_frustration 

16 

2.00 

18.00 

10.6875 

5.54640 

-.122 

.564 

Valid  N  (listwise) 

16 
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A-7  -  Experiment  2:  Robot  Control  Questionnaire  (N  =  19) 


1.  Using  the  scale  below,  please  the  training  you  received: 


1  2  3  4  5  6  7 

Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely  good 


Verbal  helped  with  initial  understanding  of  tasks.  1 

Practice  allowed  me  to  understand  how  to  use  equipment.  1 

2.  What  were  the  easiest  and  hardest  training  tasks  to  learn? 

Comments  No.  of  Responses 


Easiest 

How  to  move  forward.  1 

Using  head  motion  control.  1 

Headset.  2 

Head  tracking  the  video.  1 

Moving  my  head  with  the  mono  headset.  1 

Controlling  the  movement  of  the  robot.  4 

How  to  maneuver  camera  both  with  head  motion,  as  well  as  manual  (joystick).  1 

Using  the  remote  to  control  the  robot.  1 

Moving  the  robot  as  an  entire  unit.  1 

Concept  behind  controls.  1 

Using  head  motion  to  adjust  point  of  focus  was  extremely  intuitive.  It  was  easy  to  1 

determine  direction  of  sound  in  stereo  mode. 

Controlling  with  joystick  (similar  to  play  station  joystick  and  controlling  much  like  a  1 
modem  first  person  shooter  with  left  sick  being  motion  and  right  being  camera). 

Using  head  tracking. 

Utilizing  the  antennas  on  the  robot  to  avoid  hitting  walls  or  objects.  1 

Working  the  joystick  and  headset  in  unison  was  much  easier  than  I  expected  and  it  1 

makes  controlling  the  robot  a  breeze. 

Visual  capabilities  were  the  best  part  about  the  training  task.  1 

The  verbal  explanations  given  by  the  staff  members.  They  broke  it  down  step  by  step  1 
which  was  really  easy  to  understand. 

Hardest 

None.  1 

Spatial  awareness  forced  me  to  always  look  down  at  the  antenna  on  either  side  or  the  1 
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Comments  No.  of  Responses 

red  pointer,  but  got  easier  as  time  went  on. 

Pinpointing  the  exact  place  the  noise  is  coming  from.  3 

Being  asked  not  to  move  your  head;  to  keep  head  still.  3 

Not  moving  my  head  when  controlling  the  head  movements  via  the  joystick  took  a  few  1 
sessions  to  get  down  completely. 

Looking  around  the  joystick  without  moving  head.  1 

Moving  the  camera  by  joystick  and  trying  to  move  the  robot  at  the  same  time.  1 

Joystick  with  two  controllers.  1 

Movement  was  difficult  at  times.  1 

Took  a  few  minutes  to  become  acclimated  to  the  robot’s  turning  radius,  particularly  1 
the  clearance  required  on  the  back  side  of  the  robot.  Adjusting  to  point  of  focus  with 
a  joystick  was  also  a  challenge. 

Making  tight  turns.  2 

Control  of  the  robot  was  a  little  difficult.  1 

Rotating  the  head  of  the  robot.  1 

Controlling  robot  from  3  person  perspective.  1 

Learning  the  robot’s  physical  dimensions  intuitively,  especially  the  clearance  in  the  1 
back. 

General  Comments 

Awesome  training;  enjoyed  using  the  technology.  1 

Very  thorough  instruction  was  given.  1 

I  learn  best  by  doing,  so  getting  my  hands  on  the  controls  was  the  best  way  to  leam  1 
how  to  maneuver  the  robot. 

Very  easy  to  pick  up  if  the  soldier  is  familiar  with  modern  FPS  videogames  because  1 
controls  are  similar  for  both  muscle  memory  on  joystick  and  the  way  the  camera  is 
controlled. 

Solid  platform  that  would  work  well  with  the  right  chassis  and  extension  arms  for  1 

control. 

Need  more  rear  vision  for  maneuverability.  1 

It’s  nearly  impossible  to  discern  if  a  noise  is  coming  from  above  or  below.  1 

Difficult  to  look  and  maneuver  robot  in  a  fluid  motion,  the  time  to  process  the  visual  1 
obstacles  (walls,  charts,  etc.(  to  the  maneuver  of  the  robot. 
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Control  Conditions 


A  -  Telepresence  (Stereo  vision  3D  audio  &  Head  tracking) 

B  -  Mono/Head  Tracking 
C  -  Mono/Joystick 

1.  Using  the  scale  below,  please  rate  how  easy  or  difficult  it  was  to  perform  each  of  the  following 
tasks  with  each  robot  control  condition  above. 

1  2  3  4  5  6  7 

Extremely  difficult  Very  difficult  Difficult  Neutral  Easy  Very  easy  Extremely  easy 


MEAI 

N  RESP< 

DNSE 

A 

B 

C 

Move  the  robot  in  the  desired  direction 

6.11 

5.74 

4.63 

Avoid  obstacles 

5.42 

5.00 

3.95 

Turn  around 

5.32 

4.95 

4.53 

Maneuver  around  corners 

5.42 

5.11 

4.47 

Back  up 

5.11 

4.74 

4.58 

Hear  the  target  sound  (sound  volume) 

5.47 

4.58 

4.11 

Locate  target  objects  by  sound  (sound  location) 

5.47 

3.89 

3.26 

Locate  target  objects  by  sight 

6.00 

5.79 

4.89 

Find  targets  quickly 

5.42 

4.79 

3.89 

Know  your  location  in  the  room 

5.95 

5.89 

5.53 

Remember  target  locations 

5.63 

5.47 

5.21 

Move  quickly 

5.47 

4.84 

4.26 

Maneuver  probe  close  to  the  IED  object  (when  requested) 

5.63 

5.32 

4.74 

2.  What  difficulties  did  you  experience  in  performing  the  target  locating  task? 

Comments  No.  of  Responses 


A  -  Telepresence 

Easiest  for  me  because  I  could  hear  where  the  sound  was  coming  from.  1 

In  the  stereo  vision  mode,  it  was  difficult  to  look  around.  1 

C  -  Mono/Joystick 

The  joystick  made  it  a  lot  harder  to  maneuver  the  robot,  as  well  as  see  to  turn  corners. 

Using  the  joystick  to  control  sight  and  movement.  1 

General  Comments 

Head  tracking  was  easy  to  control,  but  required  more  physical  exertion  than  joystick  1 
control,  which  was  both  easy  and  effortless. 

After  practicing,  it  was  quite  easy  apart  from  maintaining  special  awareness  of  the  1 

robot. 

Finding/locating  the  task.  1 

Since  some  items  are  close  to  each  other,  I  find  it  a  little  hard  to  identify  the  exact  3 

object  that  is  making  the  noise. 
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Comments  No.  of  Responses 


The  volume  was  very  low,  it  seemed.  Sometimes  I  would  have  to  guess  which  1 

direction  the  sound  was  coming  from  and  then  adjust  depending  on  whether  the 
volume  was  getting  louder  or  quieter  to  decide  if  I’m  going  in  the  right  direction. 

Hearing  where  the  target  sound  was  coming  from  with  objects  that  were  close  1 

together.  For  example,  in  the  room  with  the  landmines,  the  objects  were  close 
together  so  it  was  difficult  to  identify  which  particular  object  was  projecting  sound. 

In  mono-sound  modes,  it  was  difficult  to  tell  which  direction  the  sound  was  coming  5 
from. 

I  had  the  most  difficulty  with  the  mono-sound  and  the  joystick  movement  as  opposed  1 
to  the  head  tracking. 

When  using  the  mono/joystick,  I  sometimes  forgot  that  where  I  was  looking  wasn’t  1 
necessarily  where  the  robot  was  facing. 

I  could  not  tell  which  target  was  making  an  audio  indicator.  1 

Moving  the  head  with  the  joystick  slowed  my  down  somewhat.  1 

Joystick  was  much  more  difficult  than  using  the  head  sensor.  1 

I  forgot  about  the  mortar,  soda  cans,  and  some  other  IEDs  so  I  kept  choosing  the  more  1 
obvious  ones. 

Hard  to  determine  where  IED  is  without  Telepresence,  up/down  Joystick  would  be  1 
easier  if  inverted  (like  flight  control). 

Turning  around  was  somewhat  difficult  without  bumping  the  rear  of  the  robot.  1 

3.  Please  use  the  following  scale  to  rate  the  robot  control  system  features. 

1  2  3  4  5  6  7 

Extremely  bad  Very  bad  Bad  Neutral  Good  Very  good  Extremely  good 


DISPLAY/CONTROL  CHARACTERISTICS 

MEAI 

\  RESP< 

3NSE 

A 

B 

C 

Resolution  (clarity)  of  the  display 

5.33 

5.28 

5.11 

Precision  of  camera  movement  control 

5.44 

5.22 

5.11 

Sensitivity  of  camera  movement  control 

5.17 

4.94 

4.67 

Time  lag  of  camera  movement  control 

4.83 

5.06 

4.88 

Controlling  camera  and  movement  at  the  same  time 

5.72 

5.67 

4.53 

Capability  to  hear  sound  (volume) 

5.50 

5.17 

4.83 

Capability  to  locate  sound  (direction) 

5.33 

4.00 

3.33 

Overall  ease  of  use 

5.94 

5.50 

4.72 

Adequacy  of  this  control  unit  for  tele-operating  a  robot 

5.67 

4.94 

4.61 

Overall  assessment  of  this  control  unit 

5.78 

5.00 

4.39 

4.  Please  indicate  how  much  each  of  the  following  features  would  contribute  to  a  recon  mission. 


1  2  3  4 

Extremely  bad  Very  bad  Bad  Neutral 


5  6 

Good  Very  good 


Extremely  good 
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MEA1 

V  RESP< 

3NSE 

A 

B 

C 

3-D  audio 

6.56 

NA 

NA 

Stereo  vision 

6.22 

NA 

NA 

Head-tracked  camera 

6.06 

NA 

NA 

Mono  audio 

3.67 

NA 

NA 

Mono  vision 

4.11 

NA 

NA 

Joystick  camera 

4.56 

NA 

NA 

Comments  No.  of  Responses 

A  -  Telepresence 

The  easier  it  is  to  control  the  robot,  the  more  success  it  will  have  in  the  field.  1 

Definitely  more  effective  with  this  one.  1 

Control  was  very  intuitive,  but  I  found  myself  turning  my  head  in  joystick  mode  and  1 
vice  versa. 

I  found  that  while  the  stereo  sound  was  indispensible  to  “hearing”  the  target,  stereo 
video  tended  to  lag  when  I  was  trying  to  go  quickly  or  move  my  head  or  the  robot. 

Almost  made  me  crash  a  few  times. 


Head  tracking  has  a  very  short  learning  curve;  with  an  hour’s  practice,  it  becomes  1 

second  nature. 

General  Comments 

Everything  works  great  except  for  two  things:  the  top  of  the  head  gear  puts  pressure  1 
on  my  head  and  movement  of  the  camera  gives  me  slight  motion  sickness. 

Joystick  camera  is  better.  No  lag.  It  offers  more  precision.  1 

Joystick  control  has  a  much  longer  learning  curve  for  a  non-gamer.  1 

Head  tracking  is  not  a  useful  work  tool  unless  it  is  integrated  with  a  Kevlar  helmet.  1 

I  think,  for  me  at  least,  stereo  sound  and  mono  video  would  be  the  best  combination,  if  1 
possible. 

A  zooming  feature  for  the  camera  would  have  been  nice,  especially  useful  on  the  1 

second  test  with  the  randomly  placed  items.  4  minutes  in  locating  items  from  a 
distance  instead  of  wasting  time  maneuvering  towards  it  to  identify  it. 

I’d  like  to  try  the  robot  using  stereo  vision  and  audio,  but  control  the  camera  1 

movements  via  remote  control.  I  was  finding  it  easier  to  maneuver  the  robot  using 
directional  pad  camera  control,  especially  around  corners. 


5.  Using  the  scale  below,  please  rate  how  easy  or  difficult  it  was  to  perform  the  following: 

1  2  3  4  5  6  7 

Extremely  difficult  Very  difficult  Difficult  Neutral  Easy  Very  easy  Extremely  easy 


MEAI 

N  RESP< 

3NSE 

A 

B 

C 

Maneuver  probe  close  to  the  IED  object  (when  requested) 

5.79 

5.00 

4.53 
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Comments 


No.  of  Responses 


A  -  Telepresence 

It  seemed  easier  to  get  closer  to  an  IED  using  the  stereo  visual  display  based  on  depth  1 
perception.  What  helped  was  the  pointing  device  located  in  the  center  of  the  robot. 

C  -  Mono/Joystick 

Joystick  took  longer  to  adjust  than  vision  of  the  robot.  1 

Joystick  was  difficult  to  maneuver.  1 

General  Comments 

Pretty  straightforward;  simple.  1 

When  the  probe  was  over  top  an  object  (box),  I  couldn’t  tell  if  the  body/wheels  would  1 
hit  that  box  before  the  probe  reached  the  target. 

It  was  hard  to  maneuver  the  robot  close  to  probe  the  IED,  but  it  does  take  time  to  do  1 
so. 

Unable  to  see  or  know  exactly  what  the  rear  of  the  robot  was  doing.  1 

The  antenna  and  red  probe  would  have  been  much  more  useful  if  they  were  in  the 
centered/zeroed  sight  because,  especially  with  the  joystick,  it  slowed  me  down  to 
always  be  looking  down  at  them  and  also  made  me  lose  perception  of  the  centered 
sight.  It  got  easier  only  because  I  had  the  room  removed  with  the  layout  and  IED 
placement  but  if  it  was  switched  or  random,  it  would  have  really  messed  me  up. 


6.  Which  control  condition  did  you  prefer?  Why? 


Number  of  Responses 

Telepresence 

Mono/Head  Tracking 

Mono/Joy  stick 

No  Response 

13 

2 

3 

1 

Comments  No.  of  Responses 


A  -  Telepresence 

Just  seemed  easier  to  use.  2 

Easier  for  sight.  1 

Easier  to  locate  target.  1 

Firing  arm  is  free  for  use.  1 

Natural.  1 

The  most  impressive,  allowing  a  quicker  assessment  of  the  situation  and  greater  ease  1 
of  maneuver. 


Easier  to  locate  and  identify  objects  while  being  able  to  move  your  head  (as  part  of  the  1 
camera/stereo  vision).  I  felt  the  3D  audio  was  very  precise  and  so  was  the  head 
tracking  feature. 

3D  audio  helped  a  lot  for  target  identification  when  multiple  units  were  in  the  same  2 
area. 

3D  audio  helped  to  locate  the  direction  of  the  sound  a  lot  faster.  1 

Because  I  naturally  was  inclined  to  move  my  head  with  the  3D  view.  With  video  1 

games,  I  was  used  to  the  joystick,  but  the  television  for  that  is  further  away  and  less 
impressive,  which  this  system  it  was  much  easier  to  use  the  head  tracking. 
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Comments  No.  of  Responses 

I  didn’t  notice  an  appreciable  difference  in  tenns  of  the  stereo  vision  increasing  my  1 
depth  perception.  This  may  have  been  mainly  attributable  to  the  poor  quality  of  the 
display. 

Felt  the  most  precise.  I  could  move  quickly  and  fluidly  through  the  course.  1 

Easiest  overall  control  of  box  and  easiest  ability  to  listen  to  where  the  sound  was  2 

coming  from. 

The  head  tracking  was  extremely  helpful  in  the  overall  maneuvering  of  the  robot.  1 

It  combined  the  head  movement  along  with  the  stereo  audio.  That  was  a  huge  plus. 

B  -  Mono/Head  Tracking 

Felt  like  I  was  in  control  of  the  robot  more  while  in  mono/head  tracking.  Could  focus  1 
more  on  the  task  at  hand. 

Felt  I  could  find  the  target  quicker.  1 

C  -  Mono/Joystick 

I  preferred  maneuvering  the  robot  using  the  joystick  camera.  1 

Easier  to  use  joystick  to  move.  1 

A  lot  easier  after  being  on  the  machine  for  over  30  minutes.  1 

It  gives  me  almost  a  360°  view  of  the  environment  and  it  wouldn’t  have  to  turn  my  1 
head  often  which  decreases  my  motion  sickness  that  I  was  feeling  during  the  test. 

General  Comments 

I  want  stereo  sound  and  joystick  controls.  1 

The  mono  stereo  was  more  near  or  far  not  left  or  right.  Confusing  (mono)  when  more  1 
than  one  target  was  in  same  location. 

7.  What  are  some  suggestions  for  improving  the  robot  control  system? 

Comments  No.  of  Responses 

Outstanding  equipment!  1 

More  practice  to  make  sure  it  would  work  in  combat.  1 

Increase  resolution  of  the  camera  and  improve  response  time  in  vision  and  movement.  1 
Movements  were  also  somewhat  jerky. 

Try  to  make  a  clearer  picture  and,  if  possible,  a  smaller  robot  would  be  more  effective  1 
as  well. 

Make  robot  movement  speed  change/respond  to  how  far  forward  joystick  is  pressed.  1 

For  objects  that  are  close  together,  maybe  some  way  to  really  make  that  particular  1 

object  stand  out  through  sound. 

Increase  the  sound/hearing  capability.  I  had  the  hardest  time  pinpointing  the  exact  1 
location  to  move  in  because  I  sometimes  could  not  tell  where  the  sound  was  coming 
from. 

Have  a  rear  facing  camera  that  you  can  switch  to  for  better  maneuverability.  1 

Use  the  stereo  vision  and  3D  audio  with  head  tracking.  1 

For  3D  and  head  tracking,  maybe  we  could  be  able  to  switch  off  one  ear  or  the  other  in  1 
order  to  better  identify  where  the  target  is  coming  from  or  have  volume  sensitivity 
between  the  robot  ears  and  the  wearer’s  ears,  i.e.,  if  the  sound  is  coming  from  the  left 
have  it  reflected  from  the  left  ear  piece  and  vice  versa. 
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Comments  No.  of  Responses 

Make  the  head  tracking  system  less  bulky.  1 

The  robot  is  fine;  however,  decrease  the  size  of  the  robot  to  make  it  easier  to  maneuver  1 
in  tight  areas.  Make  headset  for  robot  lighter  because  after  awhile,  it  puts  pressure  on 
the  top  of  the  head  which  in  turn  gives  me  a  headache,  plus  the  motion  sickness. 

Utilize  the  clip  on  the  Kevlar  helmet  to  place  the  VR  goggles  where  NVGs  typically  1 
go- 

The  head  tracking  pinches  the  brain  and  gave  me  a  headache.  1 

Find  a  display  system  which  does  not  induce  headaches/motion  sickness.  Display  1 

should  also  do  a  better  job  of  utilizing  stereo  vision  capabilities. 

A  more  comfortable  helmet  would  be  nice,  but  it  would  also  be  helpful  to  possibly  1 
include  more  peripheral  vision;  though  this  would  require  some  ingenuity. 

For  those  wearing  glasses,  the  headset  moves  the  glasses  and  makes  vision  or  focusing  1 
a  small  problem. 

Maybe  not  a  360°,  but  more  of  a  turning  radius  for  vision.  1 

Mono/Joystick  should  have  up/down  inverted  like  flight  stick.  1 

A  less  bulky  body  would  help  with  maneuvering.  1 

More  range  on  the  axis  of  the  camera.  1 

Ability  to  see  down/left/right  more.  1 

With  the  joystick,  an  XBOX  360  controller  set-up  would  be  easier  for  both  control  of  1 
the  robot  and  the  camera  than  a  play  station  style  controller. 

A  more  comfortable  headset  would  place  less  stress  on  the  crown  and  make  it  easier  to  1 
use  the  head  tracking  effectively  and  for  longer  periods  of  time. 

Crash/collision  sensors.  1 

A  heads-up  display.  1 


8.  Please  check  any  of  the  following  conditions  that  you  may  have  experienced  during  this  trial. 


No.  of  Responses 

Eyestrain 

8 

Tunnel  vision 

0 

Headaches 

8 

Motion  sickness 

5 

Nausea 

5 

Disorientation 

0 

Dizziness 

4 

Competition  between  eyes  for  vision  of  the  different  scenes 
at  which  they  are  looking 

4 

Other 

7 

Comments  No.  of  Responses 


Same  discomfort  from  weight.  1 

The  headset  hurt  my  head  (too  heavy/hard/tight)  and  created  discomfort.  4 

Fatigue.  1 

Hands  started  to  tingle  with  the  increase  of  above  symptoms  (headaches,  motion  1 
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Comments  No.  of  Responses 

sickness,  nausea,  dizziness). 

I  was  only  able  to  do  tasks  in  groups  of  3  before  nausea  had  set  in.  After  each  break,  I  1 
was  able  to  resume,  but  the  nausea  came  back  at  a  faster  rate  each  time. 

Headaches  may  be  due  to  seeing  double  during  the  breaks.  Also,  there  was  some  lag  at  1 
one  point. 
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Intentionally  lelt  blank. 
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Appendix  B.  National  Aeronautics  and  Space  Administration  Task  Load 

Index  (NASA-TLX)  Mental  Workload 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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The  NASA-TLX  was  administered  using  a  computer-based  version.  First,  Soldiers  were  asked  to 
rate  their  experience  with  the  controller,  for  the  following  task  demand  factors.  Afterward, 
Soldiers  completed  paired  comparisons,  choosing  from  each  pair  the  aspect  which  contributed 
more  to  their  workload  experience. 

NASA-TLX  Mental  Workload  Rating  Scale 

Please  place  an  “X”  along  each  scale  at  the  point  that  best  indicates  your  experience  with  the 
robot  controller  you  just  used. 


Mental  Demand:  How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  deciding, 
calculating,  remembering,  looking,  searching,  etc)?  Was  the  mission  easy  or  demanding,  simple  or 
complex,  exacting  or  forgiving? 

Low  i _ | _ | _ | _ | _ | _ | _ | _ | _ i _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ I  High 


Physical  Demand:  How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning, 
controlling,  activating,  etc.)?  Was  the  mission  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous, 
restful  or  laborious? 

Low  I _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ I  High 


Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the 
mission  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

Low  I _ | _ | _ | _ | _ | _ | _ | _ | _ | _ I _ | _ | _ | _ | _ | _ | _ | _ | _ | _ I  High 


Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  mission?  How 
satisfied  were  you  with  your  performance  in  accomplishing  these  goals? 

Low  I _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ I  High 


Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 

Low  I _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ | _ I  High 


Frustration:  How  discouraged,  stressed,  irritated,  and  annoyed  versus  gratified,  relaxed,  content, 
and  complacent  did  you  feel  during  your  mission? 

Low  I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I _ I  High 
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Definition  of  Task  Demand  Factor 


Mental  demand 

How  much  mental  and  perceptual  activity  was  required  (e.g.,  thinking,  deciding,  calculating, 
remembering,  looking,  searching,  etc.)?  Was  the  task  easy  or  demanding,  simple  or  complex, 
exacting  or  forgiving? 

Physical  demand 

How  much  physical  activity  was  required  (e.g.,  pushing,  pulling,  turning,  controlling, 
activating,  etc.)?  Was  the  task  easy  or  demanding,  slow  or  brisk,  slack  or  strenuous,  restful  or 
laborious? 

Temporal  demand 

How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  tasks  or  task 
elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 

Performance 

How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  task  set  by  the 
experimenter  (or  yourself)?  How  satisfied  were  you  with  your  performance  in  accomplishing 
these  goals? 

Frustration  level 

How  insecure,  discouraged,  irritated,  stressed  and  annoyed  versus  secure,  gratified,  content, 
relaxed  and  complacent  did  you  feel  during  the  task? 

Effort 

How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 
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NASA-TLX  Mental  Workload  Rating 


Soldiers  performed  the  following  paired  comparisons  within  a  computer-administered  version  of 
the  NASA-TLX. 

For  each  of  the  pairs  listed  below,  circle  the  scale  title  that  represents  the  more  important 
contributor  related  to  operating  the  robot  controller. 


Mental  Demand 

or 

Physical  Demand 

Mental  Demand 

or 

Temporal  Demand 

Mental  Demand 

or 

Performance 

Mental  Demand 

or 

Effort 

Mental  Demand 

or 

Frustration 

Physical  Demand 

or 

Temporal  Demand 

Physical  Demand 

or 

Performance 

Physical  Demand 

or 

Effort 

Physical  Demand 

or 

Frustration 

Temporal  Demand 

or 

Performance 

Temporal  Demand 

or 

Frustration 

Temporal  Demand 

or 

Effort 

Performance 

or 

Frustration 

Performance 

or 

Effort 

Frustration 

or 

Effort 
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Appendix  C.  Informed  Consent  Form 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Informed  Consent  Form 

Army  Research  Laboratory,  Human  Research  &  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 


Title  of  Project:  Robotic  Telepresence  User  Interfaces:  Situation  Awareness  and  Perceptual 
Motor  Tasks 


Project  Number:  9MC254 

Sponsor:  Army  Research  Laboratory 

Principal  Investigator:  Linda  R.  Elliott 

Army  Research  Laboratory 

Human  Research  &  Engineering  Directorate 

RDRL-HRM-DW 

Lort  Benning,  GA.  31905 

(706)  545-9145;  linda.r.elliott@us.army.mil 

Associate  Investigators:  Elizabeth  S.  Redden  and  Rodger  Pettitt 

Army  Research  Laboratory 

Human  Research  &  Engineering  Directorate 

RDRL-HRM-DW 

Fort  Benning,  GA.  31905 

(706)  545-9142;  Rodger.Pettitt@us.army.mil 

Elizabeth.Redden@us.army.mil 


Chris  Jansen 

TNO  Defense,  Safety,  and  Security  (Human  Factors) 
Kampweg  5  Soesterberg 
P.O.  Box  23,  The  Netherlands 
chris.iansen@tno.nl 
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Informed  Consent  Form 

Army  Research  Laboratory,  Human  Research  &  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 


You  are  being  asked  to  join  a  research  study.  This  consent  form  explains  the  research  study  and 
your  part  in  it.  Please  read  this  form  carefully  before  you  decide  to  take  part.  You  can  take  as 
much  time  as  you  need.  Please  ask  the  research  staff  any  questions  at  any  time  about  anything 
you  do  not  understand.  You  are  a  volunteer.  If  you  join  the  study,  you  can  change  your  mind 
later.  You  can  decide  not  to  take  part  now  or  you  can  quit  at  any  time  later  on. 


Purpose  of  the  Study 


The  purpose  of  this  evaluation  is  to  determine  the  effects  of  two  different  robot  control  interfaces 
on  Soldier  performance  while  conducting  robotic  search  tasks. 


Procedures  to  be  Followed 


The  study  will  be  conducted  using  two  robot  control  devices:  regular  teleoperation  and  enhanced 
Telepresence  teleoperation.  After  training  on  the  operation  of  the  controllers,  you  will 
be  asked  to  conduct  two  building  and  route  reconnaissance  missions  using  an  unmanned  ground 
vehicle.  Effectiveness  of  robot  operation  will  be  evaluated  based  on  objective  performance  data, 
data  collector  observations,  and  self-report  questionnaires. 

You  will  be  asked  to  complete  a  demographics  questionnaire  regarding  your  military  training 
and  experience. 

Trainers  from  the  Army  Research  Lab  and  TNO  Netherlands  will  provide  a  course  on  the  use  of 
the  robot  control  systems.  The  training  course  will  include  hands-on  exercises.  You  will  be  asked 
to  complete  a  questionnaire  on  the  adequacy  of  the  training. 

Discomforts  and  Risks 

The  risks  that  will  be  encountered  during  this  investigation  are  typical  of  the  risks  encountered 
performing  training  and  duties  pertaining  to  your  military  occupational  specialty.  These  risks 
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Informed  Consent  Form 

Army  Research  Laboratory,  Human  Research  &  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 


include  physical  fatigue  and  work-related  injury.  There  is  also  a  potential  risk  of  motion 
sickness.  You  should  inform  your  unit  leadership  or  the  experiment  personnel  if  you  experience 
any  discomfort  or  if  you  have  any  problems  during  the  evaluation.  You  may  be  told  to  stop 
activities  until  problems  or  conditions  are  resolved. 


You  will  be  controlling  the  robots  from  inside  a  building.  Activities  will  be  suspended 
during  any  weather  conditions  that  are  inherently  dangerous  or  will  cause  evaluation 
trials  to  be  dangerous.  Water  will  be  available  and  you  will  be  instructed  to  drink  often. 
Water  breaks  will  occur  at  least  every  30  minutes  for  all  trials  that  exceed  30  minutes  in 
duration. 


In  a  site  emergency  from  any  source,  personnel  will  have  the  capability  for  radio  and  telephone 
contact  with  the  Fort  Benning  Ground  Fire  Department  and  Emergency  Medical  Service  (EMS). 
EMS  ambulance  service  is  approximately  20  minutes  away  at  Martin  Army  Hospital  and  is 
available  24  hours  a  day. 

Benefits 

You  will  receive  no  benefits  from  participating  in  the  evaluation,  other  than  the  personal 
satisfaction  of  supporting  the  Army’s  research  in  robotics  for  Soldiers. 

Duration 

Your  participation  will  last  approximately  8  hours,  from  0800  to  1700. 

Confidentiality 

Your  participation  in  this  research  is  confidential.  The  data  will  be  stored  and  secured  in  the 
offices  of  the  principal  investigator  in  a  locked  file  cabinet.  The  data,  without  any  identifying 
information,  will  be  transferred  to  a  password-protected  computer  for  data  analysis.  After  the 
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Informed  Consent  Form 

Army  Research  Laboratory,  Human  Research  &  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 


data  is  put  in  the  computer  file,  the  paper  copies  of  the  data  will  be  shredded.  This  consent  form 
will  be  retained  by  the  principal  investigator  for  a  minimum  of  three  years. 


If  the  results  of  the  experiment  are  published  or  presented  to  anyone,  no  personally  identifiable 
information  will  be  shared.  Publication  of  the  results  of  this  study  in  a  journal  or  technical  report, 
or  presentation  at  a  meeting,  will  not  reveal  personally  identifiable  information.  The  research 
staff  will  protect  your  data  from  disclosure  to  people  not  connected  with  the  study.  Officials  of 
the  U.  S.  Army  Human  Research  Protections  Office  and  the  Army  Research  Laboratory’s 
Institutional  Review  Board  are  permitted  by  law  to  inspect  the  records  obtained  in  this  study  to 
insure  compliance  with  laws  and  regulations  covering  experiments  using  human  subjects. 
Complete  confidentiality  cannot  be  promised,  particularly  if  you  are  a  military  service  member, 
because  information  bearing  on  your  health  may  be  required  to  be  reported  to  appropriate 
medical  or  command  authorities. 


We  would  like  your  permission  to  take  pictures  or  video  of  the  experimental  session.  The 
pictures  will  be  used  to  document  problems  encountered  during  the  study.  Photographic  or  video 
images  of  you  taken  by  HRED  personnel  will  not  be  identified  with  any  of  your  personal 
information  (name,  rank,  or  status),  although  the  photos  which  include  your  face  may  be 
included  in  the  evaluation  report.  Please  indicate  below  if  you  will  agree  to  allow  us  to  record 
you.  You  can  still  be  in  the  study  if  you  prefer  not  to  be  recorded. 

I  give  consent  to  be  photographed  during  this  study:  _ Yes  _ No  please  initial: _ 

I  give  consent  to  be  videotaped  during  this  study:  _ Yes  _ No  please  initial: _ 

If  you  choose  not  to  participate  in  this  evaluation,  you  can  convey  that  choice  privately  to  the 
evaluation  manager,  who  will  then  inform,  without  elaboration,  to  your  unit  leadership  that  you 
did  not  meet  the  criteria  for  participation  in  the  evaluation. 
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Informed  Consent  Form 

Army  Research  Laboratory,  Human  Research  &  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 


Contact  Information  for  Additional  Questions 


You  have  the  right  to  obtain  answers  to  any  questions  you  might  have  about  this  research  both 
while  you  take  part  in  the  study  and  after  you  leave  the  research  site.  Please  contact  anyone  hsted 
at  the  top  of  the  first  page  of  this  consent  form  for  more  information  about  this  study.  You  may 
also  contact  the  Chairperson  of  the  Human  Research  &  Engineering  Directorate,  Institution 
Review  Board,  at  (410)  278-5992  with  questions,  complaints,  or  concerns  about  this  research,  or 
if  you  feel  this  study  has  harmed  you.  The  chairperson  can  also  answer  questions  about  your 
rights  as  a  research  participant.  You  may  also  call  the  chairperson’s  number  if  you  cannot  reach 
the  research  team  or  wish  to  talk  to  someone  else. 


Voluntary  Participation 

Your  decision  to  be  in  this  research  is  voluntary.  You  can  stop  at  any  time.  You  do  not  have  to 
answer  any  questions  you  do  not  want  to  answer.  Refusal  to  take  part  in  or  withdrawing  from  this 
study  will  involve  no  penalty  or  loss  of  benefits  you  would  receive  by  staying  in  it. 

Military  personnel  cannot  be  punished  under  the  Uniform  Code  of  Military  Justice  for  choosing 
not  to  take  part  in  or  withdrawing  from  this  study,  and  cannot  receive  administrative  sanctions 
for  choosing  not  to  participate. 

Civilian  employees  or  contractors  cannot  receive  administrative  sanctions  for  choosing  not  to 
participate  in  or  withdrawing  from  this  study. 

You  must  be  18  years  of  age  or  older  to  take  part  in  this  research  study.  If  you  agree  to  take  part 
in  this  research  study  based  on  the  information  outlined  above,  please  sign  your  name  and  the 
date  below. 


You  will  be  given  a  copy  of  this  consent  form  for  your  records. 
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Informed  Consent  Form 

Army  Research  Laboratory,  Human  Research  &  Engineering  Directorate 
Aberdeen  Proving  Ground,  MD  21005 


This  consent  form  is  approved  from  19  October  2009  to  18  August  2010. 
Do  not  sign  after  the  expiration  date  of  18  August  2010 


Participant’s  Signature 


Date 


Participant’s  Printed  Name 


Signature  of  Person  Obtaining  Consent 


Date 


Printed  Name  of  Person  Obtaining  Consent 
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NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  HRR 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LT 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  D 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 

320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

2  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI 

T  DAVIS 

J  HANSBERGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  DG  K  GUNN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 

AWC  FIELD  ELEMENT 
RDRL  HRM  DI  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DQ  MR  FLETCHER 
NATICK  SOLDIER  CTR 
AMSRD  NSC  WS  E  BLDG  3  RM  343 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303A 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  11  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CN  R  SPENCER 
DCSFDI  HF 

HQUSASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  E  REDDEN 
6450  WAY  ST 
BLDG  2839  RM  310 
FORT  BENNING  GA  31905-5400 
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1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

5  DIR  USARL 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM 

P  S  A V AGE-KNEPS  HIELD 
RDRL  HRM  A 
D  HARRAH 
RDRL  HRS  D 
B  AMREIN 
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